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EXECUTIVE SUMMARY 


During the summer of 2008, the Minnesota Department of Education contracted with 
Hezel Associates, LLC, to conduct an external evaluation of the Q Comp 
implementation. The evaluation team has adopted a formative-summative approach to 
the investigation of Q Comp impact and sustainability. This approach has allowed us 
to study how Q Comp works and whether it works as intended and also to determine 
any preliminary impact that Q Comp has had on outcomes at various levels (e.g., 
district, school, teacher, student) while implementation occurs. This final evaluation 
report describes the research activities undertaken to date; findings, commendations 
and recommendations that have resulted from our research; and suggestions for further 
research in the future. 


Hezel Associates’ framework for the evaluation of Q Comp was designed to determine 
whether the system is understandable, fair and credible, appropriately linked to desired 
behaviors and improved student performance and an efficient investment of resources. 
To do this, the Hezel team has considered each of Q Comp’s components individually 
as well as in concert with one another: career ladder/advancement options, job- 
embedded professional development, teacher evaluation, performance pay, and 
alternative salary schedule. 


A key component of the evaluation is assuring that we have sought multiple types of 
data from multiple sources. Our methods and findings sections detail these activities 
and learnings, and sustain our commendations and recommendations for the future of 
Q Comp. Here, we present the commendations and recommendations for Q Comp that 
we have been able to cull from our efforts during the fall of 2008 to the winter of 2009, 
organized by statewide and site-specific aspects of program implementation and 
impact. 


A. COMMENDATIONS 


1. Statewide implementation of Q Comp 


e The perception of Q Comp’s impact on schools has overall been quite positive. 

e Teachers, mentors, and coaches share a common view that the financial gains 
behind Q Comp are not a driving motivator for program participation. 

e Minnesota’s teachers seem to be more supported in their role of improving 
students' educational achievement since Q Comp’s inception, compared to non- 
Q Comp teachers, though we cannot link these trends statistically. 

¢ QComp emphasizes all teachers growing professionally, which is consistent 
with MDE’s goals for the school improvement process. 
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The flexible nature of Q Comp has been helpful to empowering local districts 
and schools with resources they can use to undergo the school improvement 
process. 


2. Districtwide implementation of Q Comp 


The perception that instructional practices have improved under Q Comp is 
widespread within Q Comp schools. 

Schools that have implemented Q Comp tend to view the program as an 
integration of each of the five components, and not simply a performance pay 
system. 

Onsite and meaningful professional development that is integrated into teachers’ 
schedules under Q Comp facilitates teachers’ participation in regular and more 
substantive, as well as school improvement, activites. 

Teachers are sharing and collaborating around student needs and instructional 
practices more than they ever have since Q Comp was implemented. 
Participating schools praise Q Comp for providing a unifying focus and 
framework for collaborating around instruction, planning and professional 
development. 

Teachers attribute greater consistency in the way that expectations for students 
are set and in the teaching strategies that are being used to their participation in 
Q Comp. 

Generally, observations in Q Comp schools are viewed as constructive, whereas 
in non-Q Comp schools, observations are perceived to be more evaluative. 

In some Q Comp schools, there has been an overall shift from administrative 
decision making to teacher decision making, which administrators and teachers 
view favorably. 

Teachers from Q Comp schools are generally familiar with their career ladder 
systems, or believe this information is widely available and easily accessible. 
When Q Comp is implemented in schools, the following set of conditions best 
predict student achievement to increase: (1) When school administrators feel 
that their teachers consider Q Comp to be successful in their school,(2) When 
teachers feel that someone other than the principal is responsible for conducting 
Q Comp teacher evaluations/ observations, (3) When standards-based lessons are 
not the main topic of professional development activities and discussions, but 
other topics are addressed, (4) When teachers feel that the addition of multiple 
career paths in their school will encourage them to remain in the teaching 
profession longer. 

There is a significant and positive relationship between the number of years a 
school has been implementing Q Comp and student achievement and the 
number of years a school is in Q Comp with student academic achievement. 
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B. RECOMMENDATIONS 


1. Statewide implementation of Q Comp 


e Target audiences for Q Comp awareness campaigns should include both districts 
and schools. 


O 


Strategy: Provide information about the variety of ways that districts and 
schools have designed programs to meet the requirements of specific 
elements of Q Comp. 

Strategy: Better utilize the Q Comp informational conference offered by MDE 
each January. 

Strategy: Provide information about what Q Comp does - and does not - 
involve. 

Strategy: Encourage districts and schools that are interested in Q Comp 
participation to speak with or observe a school site that is already successfully 
participating. 

Strategy: Highlight the benefits of Q Comp participation to school and 
district administrators. 

Strategy: Indicate that changes in culture and thinking can be positive. 
Strategy: Increase publicly available success stories about Q Comp 
implementation and impact. 


2. Districtwide implementation of Q Comp 


e Monitor and respond to teachers’ experiences with Q Comp. 

e Show teachers how to manage the aspects of Q Comp that are viewed as 
cumbersome. 

e Support each district in clarifying how Q Comp complements other district 
initiatives. 
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INTRODUCTION 


During the summer of 2008, the Minnesota Department of Education contracted with 
Hezel Associates, LLC, to conduct an external evaluation of the Q Comp 
implementation. The evaluation team has adopted a formative-summative approach to 
the investigation of Q Comp impact and sustainability. This approach has allowed us 
to study how Q Comp works and whether it works as intended and also to determine 
any preliminary impact that Q Comp has had on outcomes at various levels (e.g., 
district, school, teacher, student) while implementation occurs. This final evaluation 
report describes the research activities undertaken to date; findings, commendations 
and recommendations that have resulted from our research; and suggestions for further 
research in the future. 


A. OVERVIEW OF THE EVALUATION 


Hezel Associates’ framework for the evaluation of Q Comp was designed to determine 
whether the system is understandable, fair and credible, appropriately linked to desired 
behaviors and improved student performance and an efficient investment of resources. 
To do this, the Hezel team has considered each of Q Comp’s components individually 
as well as in concert with one another: 


e Career Ladder/Advancement Options: are designed to provide interested and 
qualified teachers the opportunity to take on leadership roles in the district and 
share their expertise with their colleagues while still retaining a primary role in 
student instruction. 


¢ Job-embedded Professional Development: align professional development 
with schoolwide student achievement goals on a standardized assessment and 
provide teachers with time for collaboration and collegiality during the school or 
teacher contract day. 


e Teacher Evaluation/Observation: help teachers show continuous improvement 
in instructional skills through formative teacher evaluation/ observation focused 
on a rubric and provided by a team of evaluators/observes. This process must 
occur at least three times per year for all teachers and must include at least two 
different evaluations/ observers. 


e Performance Pay: A system for awarding teachers, additional compensation 
based on attaining various performance indicators with at least 60 percent of any 
compensation increase focused on three factors: schoolwide student 
achievement gains on a standardized assessment; measures of student 
achievement such as classroom, grade level, or team goals; and teacher 
evaluation/ observation results. 
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e Alternative Salary Schedule: reforming the traditional “steps and lanes” salary 
schedule so that any permanent base salary increases for teachers are based on 
performance factors rather than longevity/ continued employment. 


A key component of the evaluation is assuring that we have sought multiple types of 
data from multiple sources. Our methods and findings sections detail these activities 
and learnings, and sustain our commendations and recommendations for the future of 
Q Comp. 
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METHODS 


A. RESEARCH QUESTIONS AND GENERAL APPROACH 


As in all sound evaluations, a set of core questions drives the evaluation of Minnesota’s 
Q Comp initiative. The five questions that fueled the evaluation examined whether, 
how, and under what conditions Q Comp exerted a meaningful impact on the 
performance of districts/schools, teachers and students: 


1. In what ways do districts and schools statewide implement Q Comp? In what 
ways can similarities and differences in implementation be characterized? 

2. How effective are MDE’s Q Comp infrastructures and activities in supporting 
district- and school-level implementation? 

3. To what extent does Q Comp contribute to changes in student performance and 
school improvement outcomes? To what extent are changes consistent across 
districts, content areas, grade levels and other dimensions? 

4, What internal and external conditions are necessary to support the integration of 
Q Comp into schools’ existing school improvement efforts? 

5. How effective is the incremental compensation delivered through additional pay 
for career ladder positions, performance-related pay and alternative salary scales 
in terms of improving teacher quality and increasing student achievement? 


The evaluation team addressed these research questions in a series of activities, which 
unfolded logically over the course of the five month project, ranging from more general 
formative and summative evaluation activities that took place in all participating Q 
Comp districts/schools, to more specific activities that involved a selected group of Q 
Comp and non-Q Comp comparison sites. Data collection was coordinated and 
consistent across Q Comp and non-Q Comp districts/schools so that valid comparisons 
could be made and so that formative data logically informed summative activities. 
Concurrently, evaluators collected in-depth information that was unique for each Q 
Comp district/ school. 


B. RESEARCH ACTIVITIES 


Our research activities were designed to be integrated and to matrix data across 
multiple sources in order to address the inquiry areas stated in the original Q Comp 
Evaluator RFP. Table 1 shows our plan for collecting integrated data during the Q 
Comp evaluation: 
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Table 1. Matrix of Data Collection Activities by Evaluation Areas Stated in RFP 


B.1. B.2 B.3 B.4._ B.5. 
o 2. oe n 
@ aot B88 Be & 
s O-Yo |S = 5 rE) o 2 >S 
a = Be ogge  eec# 02 
Description of participants £2 o Hoof fes|| o 5 2 
Ee ilseeuisegsl ea | & 
= oO 7 tr n 
SB ese F722 ce ao 
7) eo2l2sS Logs ans os 
Ww oa69 P2e2 52 @ 
Q SE SB Ease var 5 
= (ae oy ce oO 12) 
Activity 1: Preliminary Staff at MDE who are directly 
consultation with involved with Q Comp design, 
MDE support and/or implementation v v v v v 
(e.g., Q Comp application 
reviewers) 
Activity 2: focus group Up to five participants for each 
interviews with focus group interview, 
MDE’s Q Comp representing the School 
support staff Improvement (SI) Q Comp staff, v v 
SI Professional Development 
Team and staff outside the SI 
division 
Activity 3: Online surveys | One administrator and all 
of Q Comp teachers in every Q Comp v v 
implementation district/school 
Activity 4: Online surveys | All school board members and 
of community/ at least 25 percent of the 
stakeholder population in each census v v 
awareness of Q group, randomly sampled 
Comp 
Activity 5: Analysis of All students attending Q Comp 
student performance districts/schools 
data across all 60 Q 
Comp districts/ 
schools 
Activity 6: Comparative Districts, schools, 
case studies administrators, teachers, v v 
students 
6a. Follow up to All administrators and teachers 
implementation at each case site v v 
survey 
6b. Follow up to MDE | At least one randomly selected 
focus group district leader, school leader, V V y Zs 
interviews teacher leader and union 
representative at each site 
6c. Teacher focus No more than 15 randomly Y Y Y 
group interviews | selected teachers per site 
6d. Student All students of teachers 
performance data | participating in 6c. v v 
analysis 
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1. Activity 1: Consultation with MDE 


Hezel Associates’ initial face-to-face consultation with MDE on September 22, 2008 
provided a valuable context for the evaluation. The meeting attendees consisted of four 
members from the evaluation team (two in person and two over the phone) and four 
MDE staff. There was a break in the meeting to allow for an evaluator-facilitated focus 
group interviews (see Activity 2 below) and then the more general discussion resumed 
later in the afternoon. The meeting addressed the following topics: 


e An overview of Q Comp, including the history, five components, current context, 
participating sites, indicators of effectiveness and MDE’s role and level of 
involvement. 

e An overview of the evaluation, including the deliverables, timelines, client 
expectations and report purpose and audience. 

e Communication and coordination of logistics, including contacting Q Comp 
schools for data collection activities, strategies for encouraging study 
participation and obtaining databases (e.g., student data files, school contact 
information, demographic data available on the MDE web site, etc.). 


During the visit, MDE provided Hezel Associates with several data files that provided 
context for the districts (or schools) that have applied to participate in Q Comp, 
including both accepted and rejected applications. The Hezel team subsequently used 
this information to refine our evaluation design and activities. 


Following the kick-off meeting, it became evident that in order to comprehensively 
collect data from all Q Comp participants, it was necessary to expand upon the list of 60 
Q Comp “sites” (in most cases referring to districts) provided to the evaluators. In time, 
the evaluators understood that the number of schools in Minnesota with Q Comp 
experience (not counting the 2008-2009 school year) exceeded 4001. 


2. Activity 2: Focus group interviews with MDE staff 


While at the initial kick off meeting with MDE (see Activity 1), the evaluation team 
simultaneously conducted two focus group interviews with key staff involved in Q 
Comp. MDE invited attendees that were appropriate for and available to participate in 
the focus group interviews. Each focus group interview lasted approximately one hour 
and included staff from the School Improvement division (Q Comp and professional 
development team) and the Commissioner's office. One group had three attendees and 
the other had four attendees. The protocols for the focus group interviews provided a 
structure for gathering information about Q Comp including: 


1 The exact count is difficult to determine because of school closings, consolidations, or other factors. 
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e MDE staff roles and involvement (including technical support, networking 
meetings, annual conference, etc.) 

e MDE’s program expectations 

e The application process 

e Perceptions from various groups, including Q Comp and non-Q Comp schools, 
as well as other stakeholders and the community 

e Implementation successes and struggles with the five components 


The contextual information gathered during the focus group interviews (Appendix 2) 
provided an overview of Q Comp, helped to inform future protocol development and 
allowed for comparisons of program implementation to expectations of MDE staff. 


3. Activity 3: Q Comp implementation survey 


The Hezel team worked closely with MDE to develop two online surveys--one for 
school/ district-level administrators and another for teachers--containing a series of 
attitudinal questions regarding Q Comp that address three themes: general 
attitudes/information about Q Comp, collaboration, and implementation of the Q 
Comp components. The Hezel team then launched an email campaign in late-October 
2008, asking participants to complete the online surveys. 


The survey URLs were emailed to approximately 7,500 teachers and administrators 
across Minnesota’s Q Comp schools (see Table 2). Hezel Associates shared the names of 
all schools identified by MDE as established Q Comp participants (not newcomers to 
the program) with Market Data Retrieval (MDR), who identified 7, 393 appropriate 
recipients in their contact database - 337 of whom were principals/ assistant principals 
or other school-level administrators, and the remainder of whom were teachers. 
Further, the evaluation team emailed the administrator survey link to an additional 147 
district-level personnel with contact information provided by MDE. 


Table 2. Q Comp implementation survey response rates 


#Distributed | # Received Response rate 
Teachers 7056 1872 26.5% 
Administrators 484 180 37.2% 


The majority of administrators who responded to the statewide implementation survey 
were principal/school-level administrators (79.4%) who oversaw grades K-2 (67.2%) 
and grades 3-5 (68.3%). Nearly three-fourths of teacher respondents were female 
(73.6%) and most had obtained at least a Master’s degree (73.9%). Teachers have been 
employed at their current school for varying amounts of time, with half the respondents 
having spent one to nine years (50.9%) at their current school and the other half 10+ 
years (49.1%). Nearly a third of respondents taught general elementary education or 
were assigned to multiple subjects (self-contained/ elementary; 32.8%), and 18.1 percent 
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taught reading/language arts (see Table 3). All grade levels were present in the sample, 
and a correspondingly equal distribution was represented across responses; the greatest 
percentage of teachers taught at the K-2 grade level (32.1%) and the 9-12 grade level 
(33.8%). 


Table 3. Primary teaching assignment (n=1868)* 


Self-contained/Elementary 612 32.8% 
Reading/Language arts 339 18.1% 
Mathematics 205 11.0% 
Science 167 8.9% 
Social studies/History 155 8.3% 
Special education 113 6.0% 
Foreign language 28 1.5% 
Technology 16 0.9% 
Other 233 12.5% 


*Four respondents did not answer this survey question. 


After concluding the survey in mid-November, the Hezel team cleaned the data and 
calculated descriptive statistics on all closed-ended questions. All open-ended 
responses were then coded and analyzed. Cross-tabulations--including examinations 
by demographic variables--were analyzed on survey items that showed variability in 
initial descriptive responses (though responses were not weighted by any variables); 
these variables included years teaching experience, gender, grade level, subject, and 
highest degree obtained. Responses from both the administrator and teacher survey are 
embedded in the findings below, and any notable comparisons are indicated. Survey 
responses only represent Q Comp schools; non-Q Comp participants were not 
contacted. 


4. Activity 4: Community awareness survey 


To complement the statewide survey, focus group interview and student performance 
data analysis, the Hezel team developed a survey asking community stakeholders 
statewide about their knowledge and perceptions of Q Comp. Multiple Internet 
searches and feedback from MDE combined to yield 325 individuals with valid email 
addresses, including: PTA leaders, Q Comp advisory board members, Q Comp 
principals, the Minnesota School Board Association, union leaders, and the Minnesota 
Chamber of Commerce. Evaluators invited these individuals to complete the survey 
directly using a message co-signed by MDE staff, and also to forward the link to other 
relevant stakeholders. Since MDE and Hezel Associates have very little direct contact 
with these or any of Q Comp’s stakeholders, distribution of the stakeholder survey was 
somewhat limited. 


Although there is no way of determining the full reach of our survey invitations, out of 
325 direct invitations, 76 surveys were completed (23.4% response rate). Twenty four 
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respondents did not indicate an affliation. Of the remaining 52 respondents, about half 
were school administrators (53.8%) and one-quarter were parents (see Table 4). 


Table 4. Affiliation of Community Awareness Survey respondents (n=76) 


Subject Frequency Percent 

Parent 13 25.0% 
Local business person 5 9.6% 
School staff member iL 1.9% 
School administrator 28 53.8% 
District administrator 4 7.1% 
School board member 1 1.9% 
Total number respondents 52 100.0% 

for this question 
Did not respond 24 
Total number of survey 

76 
respondents 


5. Activity 5: Student performance data analysis 


To complement our focus group interview and statewide survey findings, the Hezel 
team conducted an analysis of student performance data to determine whether and to 
what extent school participation in Q Comp impacts student achievement as measured 
by the MCA-IIs. In order to gather the necessary data to address the central question of 
whether or not student achievement has been affected by different types of Q Comp 
implementation strategies, evaluators obtained’ student achievement data as well as 
teacher and administrator survey data from all Q Comp schools. MCA-II scale scores in 
reading and math for grades 3-8 and 10-11 for 2006, 2007, and 2008 were identified for 
each Q Comp school; this data was obtained from MDE’s Universal File Formats. 
Additional variables such as types of school and level of Q Comp/TAP implementation 
for each year were added to the database; the number of years in Q Comp/TAP was 
identified for each school through a computation using supplemental data in the 
database. Teacher and administrator survey data was also merged into the data set. 
There were 684,326 cases? in the database, including approximately 16,294 unique 
students associated with Q Comp schools. Data regarding Q Comp implementation 
strategies measured by the teacher and administrator online surveys described in 
Activity 3 were added to the MCA-II data provided by MDE. A total of 45 districts 
responded to both the teacher and administrator surveys. 


To answer the question regarding the effect that Q Comp strategies may have had on 
student achievement, the evaluation team conducted a series of regression analyses (see 
the glossary in Appendix 1) to determine the set of variables from the surveys that best 


2 The student achievement data was obtained from MDE’s Universal File Format. 

3 Many of the students were duplicated because there were a total of 42 permutations of students with 
different characteristics - two for subject (reading and math), seven for grade level (grades 3-8 plus grade 
10 in reading and grade 11 in math), and three for year (2006-2008). 
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predict student achievement. Each response on the surveys was used as independent or 
predictor variables, and MCA-II scale scores were used as a measure of student 
achievement, or the dependent variable. In total, there were 232 predictor variables. 
Given such a large number of predictor variables, evaluators conducted three different 
sets of regression analyses and, because students had more than one MCA-II score in 
the database due to multiple subjects, multiple years and multiple grades, evaluators 
computed a total of 36 regression equations - six for the first set of regressions, 12 for 
the second and 18 for the third. Regression analyses were done for both subject areas 
(reading and math), each of the three years of interest (2006, 2007, 2008), and for three 
grades (3, 8 and 10/11). 


The Hezel team used a stepwise regression technique so that each time a variable was 
entered into a regression equation, it was also possible to remove a variable from the 
equation if doing so maximized predictive power. Stepwise regression also allows two 
requirements for each variable as it is being considered for entry into the equation. One 
entry requirement is a simple correlation of the variable to the dependent variable, and 
the other is the change in predictive power that the variable contributes over the 
previous set of variables in the equation. 


For the first regression iteration, all 232 predictor variables were entered into the 
equation. This iteration of regression equations only involved MCA-II scores from 
2008, which produced a combination of 11 variables that contributed variance to the 
dependent variable. For the second iteration of regression equations, all 232 variables 
were again entered into regression equations, this time involving only data from 2006 
and 2007. Nine of the 11 variables from the first iteration contributed significant 
variance to MCA-II scores. These nine variables were the product of the second 
iteration. The third regression iteration used only these nine variables in regression 
equations involving both reading and math, grades 3, 8 and 10/11, and all three years. 
From those regression equations, the evaluation was able to identify four variables that 
contributed to test score variance in all three years. Bivariate Pearson correlations (see 
Appendix 1) of each of these four variables with MCA-II scores for both subject areas 
and all three levels for 2008 allowed us to determine the direction of the correlation. 


After completing the regression analyses, evaluators utilized statistical procedures to 
compare MCA-II scores of charter and public district schools. These procedures are 
called t-tests (see Appendix 1), and are used to compare the means of two independent 
groups. In addition, bivariate Pearson correlations revealed the relationship between 
student achievement and the number of years schools have been in Q Comp. Lastly, 
descriptive statistics for all demographic variables in the database were calculated, 
including ethnicity, limited English proficient status, free and reduced lunch status, and 
special education status. 
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6. Activity 6: Comparative case studies 


The Hezel team created the case study site selection methodology and protocols? after 
consulting with MDE and reflecting on our initial focus group interview data (Activities 
1 and 2; Appendix 2). The case study sites were hand-picked to represent unique 
combinations of characteristics such as geographic location, district size, number of 
years using Q Comp, etc. For increased variation, there was an expectation to have at 
least one TAP school and one charter school represented. Within these districts, 
evaluators identified and contacted schools for participation using additional school- 
level variables such as AYP status and student demographics with a goal of assuring 
variability across the sites. To the extent possible, MDE did not influence our site 
selection, except when some of our “first” choice sites did not want to participate for 
various reasons or did not respond. In the end, a blend of “first” and “second” choice 
sites participated in the case studies. After solidifying the seven Q Comp or TAP 
schools, evaluators then recruited seven comparable non-Q Comp schools, matched by 
district and school variables°. The fourteen schools the evaluators visited in November 
2008 had a fairly diverse distribution of characteristics (see Table 5), including years of 
Q Comp experience and progression through the Q Comp implementation process. 


Table 5. Characteristics of the 14 case study sites. 
Q Compl Q Comp District 


TAP® start year size’ Grades Location® 
School 1 Yes 05-06 Medium k-5 Rural-fringe 
School 2 Yes 05-06 Medium 9-12 Rural-fringe 
School 3 No n/a Medium k-5 Town-distant 
School 4 No n/a Medium 3-5 Town-distant 
School 5 Yes 06-07 Large 6-8 City large 
School 6 Yes 07-08 Large k-8 City —large 
School 7 No n/a Large k-5 City —large 
School 8 No n/a Large k-5 City —large 
School 9 Yes 06-07 Small k-4 Rural -distant 
School 10 Yes 06-07 Small 5-8, 9-12 | Rural -distant 
School 11 No nla Medium k-4 Town -fringe 
School 12 No n/a Medium 9-12 Town -fringe 


4 We did not have the benefit of using other anticipated data sources for informing case study site 
selection, such as the implementation survey results or student achievement data. There are other 
interesting variables to consider but the data is not readily accessible, such as leadership turnover, level 
of involvement of school leaders, varying amounts of change that has occurred since Q Comp started, etc. 
5 Although our original design called for four Q Comp sites and four non Q Comp sites, we later 


understood that “sites” meant districts and MDE preferred for us to visit two schools per district, not one 
school per district, except for the charter site. 

6 “Yes” indicates the school is aQ Comp or TAP school; “no” indicates they are not participating in Q 
Comp or TAP. 

7 According to MDE, a “small” district has a student population up to 1000, “medium” is 1001-10,000 and 
“large” is over 10,000. 

8 The location category was obtained from the National Center for Education Statistics. 
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School 13 Yes 05-06 Small k-5 City-small 
School 14 No n/a Small prek-6 Rural -fringe 


The 14 case study sites varied in MCA-II trends. The percentage of students who 
attained “Meets the Standards” or “Exceeds the Standards” on the MCA-II assessment 
for 2005-2006, 2006-2007, and 2007-2008 were examined for each case study site. In 
looking at the fourteen case study schools, the seven Q Comp schools were compared to 
the seven non-Q Comp schools (see Figures 1, 2, 3, and 4). On the face of it, the results 
of this comparison between Q Comp and non Q Comp case study schools do not appear 
markedly distinctive. No test of statistical differences was performed. 


By contrast, of the seven Q Comp schools, five schools demonstrated an increase in the 
percentage of students meeting or exceeding the mathematics standards, but two 
showed very little, if any, change. No test of statistical difference was applied to these 
comparisons. Note, however, that the three years of data may include some years in 
which Q Comp implementation had not yet begun for any given school. Of the seven 
non-Q Comp schools (#3, 4, 7, 8, 11, 12 and 14), three schools increased the percentage 
of students meeting or exceeding the mathematics standards, two decreased, one 
fluxuated and one showed very little change. 


Figure 1. Math proficiency over three years: Q Comp case study schools 
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Figure 2. Math proficiency over three years: Non-Q Comp case study schools 
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three of the seven Q Comp case study schools over time. The percentage increased over 
time for three of the seven non-Q Comp case study schools as well. 


Figure 3. Reading proficiency over three years: Q Comp case study schools 
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Figure 4. Reading proficiency over three years: Non-Q Comp case study schools 
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Since finding time for focus group interviews was difficult to incorporate into teachers’ 
schedules, evaluators did not necessarily know how many people could contribute to 
the case studies until they arrived. In some schools, administrators gave the teachers 
the option to attend the focus group interview during their planning time, while 
teachers were not given a choice in other sites. Some schools declined participation in 
the evaluation altogether simply because the teachers were not willing to give 
approximately 45 minutes to the study. In some sites, particularly non-Q Comp sites, 
teachers were very reluctant about giving up their meeting or preparation time to 
instead address the Q Comp evaluation. Since completely random teacher seletion was 
not feasible (we had to work around times that teachers were not in the classroom), a 
school representative was asked to create a schedule for the visiting evaluator. A 
request was made to include variation in the teacher focus group interviews, such as 
number of years teaching, grade level, and attitude towards Q Comp (do not “stack the 
deck” with teachers who all feel the same way). Ultimately, we met with 185 people 
during the fourteen site visits, consisting of 149 teachers or other non-administrators 
(i.e., specialists, nurses, social workers, psychologists, office staff, etc.) and 36 school or 
district-level administrators (such as principals, assistant principals, superintendents, 
union representatives, etc.). The relative proportion of teachers to administrators was 
very similar for Q Comp and non-Q Comp sites (see Figure 5). 
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Figure 5. Case Study Participants 
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The protocols used during site visits called on question areas and questions utilized in 
other aspects of the evaluation - in order to both ensure triangulated data and to allow 
for a deeper understanding of key areas than surveys alone could achieve. For Q Comp 
sites, evaluators asked all staff to complete the statewide implementation survey 
distributed earlier in the fall in order to gather the perspectives of individuals not 
directly involved with focus group interviews or other data collection events. A 
corresponding non-Q Comp survey was also developed and distributed to all staff in 
the non-Q Comp study sites. These survey results were used to supplement other case 
study data sources, rather than being analyzed in isolation. For each Q Comp and non- 
Q Comp site, evaluators utilized the following protocols: 


e Sitewide implementation survey 
e Teacher focus group interview 
e Administrator focus group/interview 


Once the site visits were complete, the evaluation team utilized an approach very 
similar to Strauss and Corbin’s? version of grounded theory to cull themes from the 
integrated data and to generate summaries of each site visited according to the 
evaluation’s broader inquiry areas. From each site’s individual summary of findings, 
evaluators generated broader projectwide findings, commendations and 
recommendations as presented below. 


? Strauss, A. & Corbin, J. (1998) Basics of qualitative research, Techniques and procedures for 
developing grounded theory, Second Edition. Thousand Oaks, CA: SAGE Publications. 
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Our integrated research methods and the use of non-Q Comp comparison sites allowed 
us to generate a summary of findings that calls on multiple types, sources and levels of 
data. This approach increases the validity of our findings while also raising additional 

research questions for future evaluation activities. 


A. MDE SCHOOL IMPROVEMENT ADMINISTRATIVE ACTIVITIES 


The consultation/ kick-off meeting and focus group interviews with key staff at MDE in 
late September shared information about the administrative activities provided by 
MDE’s School Improvement division. In November, the administrator and teacher 
focus group interviews during the case study site visits gave another perspective on the 
role that MDE plays in Q Comp implementation, as did the community awareness 
survey. MDE particularly wanted data concerning the district Q Comp plan approval, 
technical assistance, and annual Q Comp conference and networking meetings. 


1. The MDE perspective 


MDE plays a critical role in communicating and hosting activities related to Q Comp. 
Staff within the School Improvement division and Commissioner’s office range from 
partially (as low as 5%) to wholly (100%) devoted to Q Comp during their work day. 
MDE focus group interview participants described their roles in Q Comp, including 
drafting legislation; ensuring the law is carried out; handling application assistance, 
review, and approval; overseeing the Q Comp program and staff; monitoring 
implementation fidelity; writing RFPs; corresponding with Q Comp participants; 
helping to articulate SMART goals; and being involved with professional development 
as it relates to Q Comp. 


In the focus group interviews, MDE explained district applications can have great 
variability, as long as the five components are met. One participant explained, 


Applications will look different in each district, depending on what they do and 
what they have in place. Each district has choices. They just need to fulfill the 
requirements and meet the criteria. We get different applications every day. One 
component may look similar to another site, but each is very unique. 


When looking at initial applications, MDE is looking for rigor and strength of all five 
components. They look to see how it all fits together and how it is structured. They look 
for activities that are happening and how regularly they are happening. MDE checks 
the applicants’ staff development calendar. With regards to teacher observation, they 
look for what model is being used, how long it has been used, if it has been field tested, 
and if it has been modified. For performance pay, MDE examines at the goals and the 
amount of money for each goal. 
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With regard to program plan updates or revisions, MDE explained there is not a due 
date for the application, but it needs to be approved by October 1 of that year to 
implement. The plan updates come in any time (often times in June, July, and early 
August). Districts find they need to change their numbers. They may have more or less 
teachers in the upcoming year. There may be tweaks to the program. They re-evaluate 
each year. 


MDE shares information with interested, applying and participating districts. MDE 
offers an annual conference and networking meetings. The conference is offered in 
January and has a networking focus. The attendees consist of a mix of participating (Q 
Comp) and interested (non-Q Comp) school representatives. MDE presents topics such 
as defining the administrators’ role, professional learning communities, the five Q 
Comp components, and how to write quality plans. The purpose of the networking 
meetings is to engage schools already implementing Q Comp. Attendees learn from 
one another, address goal setting (including SMART" goals) and better recognize 
district needs. MDE explained the networking meetings are offered three or four times 
per year, are topic-driven, the attendees have more opportunities to interact and it is 
less formal than the conference. As a result, MDE recognizes the participants are setting 
more realistic goals. 


Information about Q Comp is also available on the MDE web site in the “teacher 
support” section. Additionally, MDE provides technical assistance to schools as they 
work through the application process and are articulating goals. Participating districts 
were surveyed this past year; however, that does not happen every year. MDE analyzes 
the results and then makes changes. MDE discusses the survey results with the Q 
Comp advisory committee based on many sources of information (including, but not 
limited to, surveys). 


When asked, “Which of the five Q Comp components have districts had the most 
difficulty implementing and where have they needed the most support?” several of the 
MDE focus group interview participants first indicated the monetary (such as reforming 
the salary structure and performance pay) and professional development components 
of Q Comp, followed by teacher evaluations. These components can be viewed as 
controversional and time-intensive. As one MDE participant explained, “The long-term 
problem has been the conversion to an alternative pay schedule.” A different staff 
person at MDE said, “The most difficult component of Q Comp for districts is 
professional development because they have to learn a new definition of it.” From 
MDE'’s perspective, the career ladder seems to be the easiest component to implement, 
partially because some districts already have one in place. 


10 SMART is an acronym for strategic, measurable, attainable, results-oriented and timely. 


Hezel Associates, LLC 16 


Q Comp Summative Evaluation Final Report for Public Release 


Focus group interview participants were specifically asked, “Do you foresee any needs 
for refinement or changes in the Q Comp structure or approval status?” With regard to 
potential changes to the Q Comp system, MDE feels that schools generally do a good 
job of implementing their stated goals and activities with the existing structure. A focus 
group interview participant commented, “We don’t need law changes. People need to 
get gradually used to it.” Another MDE staff member indicated: 


We should be cautious about not changing the statute right away. We want time 
to analyze what should be changed. At some point, maybe we could add 
principals to make them part of the program. We may need to better define the 
performance pay and salary schedule piece. 


In thinking about what capabilities the School Improvement team could consider 
strengthening to raise the quality of support provided to Q Comp districts, one MDE 
focus group interview participant shared: 


They [schools] need more information about good peer evaluation systems. The 
staff needs exposure and training in the evaluation processes. Teachers need to 
buy in if they don’t feel they have an objective and fair evaluation. Districts say 
they are doing professional development, but they need to learn how to link it to 
their building, classroom, and school goals. 


2. The school perspective 


Administrators at Q Comp schools typically heard about and became interested in Q 
Comp from administrators at other participating schools and, like administrators, 
teachers had heard about Q Comp from colleagues at other schools. Administrators 
and teachers heard more about Q Comp when it was first introduced in press releases 
by the governor and then less over time. According to our site visits and survey, 
administrators and teachers at non-Q Comp schools did not necessarily have accurate 
information about the responsibilities and purpose of Q Comp. 


Experiences with MDE varied greatly among case study participants - some knew what 
MDE 's role was in the administration of Q Comp, others had applied to participate in Q 
Comp, and yet others indicated they were not getting enough information to know 
what they thought of Q Comp. In particular, some teachers at non-Q Comp schools 
were scrambling to look up background information about Q Comp before meeting 
with the evaluation team during the site visit, as they did not know how to contribute to 
the discussion. Case study participants generally wanted to hear more from MDE 
about Q Comp. When asked, “How effective has MDE been in communicating the Q 
Comp initiative to administrators and teachers?” a non-Q Comp case study 
administrator said, “It has not been effectively communicated. It is not coming in emails 
or mail. I’m not hearing it as dialogue among colleagues or teachers.” When asked, 
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“Would you support implementing Q Comp in your school?” many non-Q Comp case 
study teachers said they didn’t know enough about Q Comp to have an opinion. When 
asked, “Are you satisfied with MDE’s involvement and offerings such as the annual Q 
Comp conference or other school improvement workshops?” a case study non-Q Comp 
administrator stated, “They [MDE] need[s] to ‘sell’ their offerings and workshops to 
districts that are interested, and not just those that are already participating.” 


3. The community perspective 


Respondents of the Q Comp community awareness survey, albeit very limited, 
provided some feedback about MDE’s administration of Q Comp. Close to two-thirds 
of the respondents either agreed or strongly agreed that Q Comp represents a good 
investment by MDE (60.6%) and that the level of information provided to them about Q 
Comp activities has been appropriate (67.7%). 


B. ATTITUDES ABOUT Q Comp 


1. Administrators and teachers 


At the start of the evaluation, MDE articulated several indicators of Q Comp success in 
focus group interviews and during the kick-off meeting. Some of these indicators are 
attitudinal and include: increased collaboration, increased interest in improving 
teaching practices, renewed excitement in teaching, improved school culture, increased 
focus on using student data, and improved teacher attraction and retention conditions. 
MDE’s expectations for indicators of successful Q Comp implementation were called 
upon in developing our evaluation protocols in order to determine whether these 
indicators are, in fact, appropriate, and also to determine perceptions of progress with 
respect to each indicator in Q Comp schools. Teachers and administrators at Q Comp 
schools clearly had different attitudes about Q Comp, when compared to those at non-Q 
Comp schools. 


In the surveys, more administrators viewed Q Comp positively compared to teachers, 
whereas in the seven case study sites that are current Q Comp participants, the majority 
of teachers and administrators shared many positive experiences with the focus group 
interview leaders. In a survey of Q Comp participants, administrators tended to agree 
that teachers considered Q Comp to be successful in their district, that administrators 
supported implementing Q Comp at their school/ district, and that most teachers in 
their school/ district support implementing Q Comp. When teachers were asked the 
same questions in their survey, few agreed. This teacher sample of survey respondents, 
however, appears to contradict the actual conditions in the Q Comp schools, in which 
the majority voted on Q Comp, suggesting most teachers view Q Comp positively. 
Teachers were typically neutral on whether the cost of implementing Q Comp could be 
better used for across-the-board raises for teachers and whether being a designated Q 
Comp school was a positive association for the school; administrators, however, rated 
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these statements more favorably. Further, while just over half of administrators found 
the program makes their school/ district more attractive, teachers were primarily 
unsure of this. Nevertheless, both teachers and administrators were largely neutral on 
whether Q Comp improves teacher retention (57-59 percent). 


Case study focus group interviews provided rich data about administrator and teacher 
attitudes toward Q Comp. In Q Comp schools, the majority of the administrators and 
teachers had to be in support of Q Comp before applying for and voting on 
participation in the program. Although some uncertainties existed in the beginning, 
after the program got off its feet, those with hesitations began to change their views. 
After seeing Q Comp’s evolution since implementation, a case study teacher indicated, 
“Now, Q Comp is more functional and meaningful to me than it was originally.” 


At first, the transitions and responsibilities associated with becoming a Q Comp school 
can be overwhelming or uncomfortable for teachers, but given the proper culture and 
structure within a school, teachers can be made to feel that Q Comp is “doable” and 
valuable. Within a non-judgmental, supportive, and collaborative setting, teachers 
enjoy and appreciate working at their school and this is no different for Q Comp 
schools. Some case study teachers who were coming into a new school chose to 
interview at certain schools because they participated in Q Comp. They had heard 
favorable comments from Q Comp participants or they enjoyed the Q Comp experience 
at their prior school. A principal at a participating school explained, “We all appreciate 
working here. At other schools, there is a lot of turn-over. We lost just one teacher last 
year. People really believe in what we do. Everyone takes responsibility for student 
growth - it’s ‘all’ of our students, not just “your’ students.” 


Given Q Comp’s hallmark features of continuity and regularity of professional 
development, increased teacher dialogue, the availability of mentors and coaches, and 
opportunities for self-reflection, teachers in Q Comp schools are looking at instruction 
under a new lens. One teacher described, “You walk away (from a meeting) feeling it is 
useful. You don’t question, ‘Why are we learning this?’ You don’t feel you're wasting 
time. There is buy-in that it’s being run well.” Another teacher explained: 


It’s far better than the process I’ve used before, which was filling out a 
Professional Development Plan. It was hollow. I would fill out paperwork, but 
with little feedback, it was just a piece of paper. I wasn’t given instruction or 
support. I was excited to join a TAP [Q Comp] school. I would now get more 
meaningful staff development. 


On the other hand, non-Q Comp schools portrayed mixed reactions about Q Comp. 
While some were interested in joining the program or even working on their 
applications, others didn’t know enough about Q Comp to have an opinion. Others had 
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some negative perceptions either from misinformation or teachers who were 
participating in Q Comp at other schools. 


When asked on the statewide implementation survey to identify the single most 
significant value of Q Comp, administrators and teachers both agreed that the program 
has led to greater collaboration with staff (32.2% administrators; 37.8% teachers). 
Improved professional development (27.1% administrators; 18.2% teachers) and 
improved student performance (16.4% administrators; 12.8% teachers) were also cited 
by respondents as having significant value, while fewer respondents indicated greater 
teacher pay for student performance (related to being in the Q Comp program) and 
greater professionalism for teachers as being significant values of the program (see 
Table 7). 


Table 6. Significant contributions of Q Comp, according to administrators and 
teachers. 


Percent Percent 

Administrators Teachers 

Response (n=180) (n=1833) 
Greater collaboration with staff 32.2% 37.8% 
Improved professional development 27.1% 18.2% 
Improved student performance 16.4% 12.8% 
Greater professionalism for teachers 14.7% 8.6% 

Greater teacher pay for student performance related to being in 
the Q Comp program 4.5% 10.5% 
Other 5.1% 12.2% 
2. MDE 


The survey and case study findings corroborate with MDE’s favorable perceptions of Q 
Comp, as discussed during the initial kick off meeting and MDE focus group interviews 
in September. MDE staff were asked about the general response or perception of Q 
Comp by teachers, principals, districts and other stakeholders in Q Comp districts, as 
well as by those in non-Q Comp districts. In general, MDE staff believes teachers in Q 
Comp sites view Q Comp positively. When teachers have not had control or input into 
the process, however, feelings can be more negative. Also, according to MDE, principals 
are generally supportive, though some may view Q Comp more negatively because 
they are misperceiving Q Comp to mean losing a certain amount of administrative 
control over school operations because they don’t understand their role. MDE senses 
that non-Q Comp districts may have negative attitudes about Q Comp because they 
don’t know enough about it. 


3. The community 


The Q Comp community awareness survey revealed attitudes and knowledge of Q 
Comp of the general public. Out of 76 respondents, 82.9 percent had heard of Q Comp 
and 71.1 percent knew of a school or district in Minnesota participating in Q Comp. 
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When presented with numerous statements and response options ranging from 
strongly disagree to strongly agree, the greatest percentage of survey respondents 
agreed that collaboration between teachers and administrators will increase as a result 
of Q Comp (61.6%), Q Comp can be successfully implemented in most Minnesota 
school districts (61.5%) and that Q Comp is effective (57.6%). The majority of 
respondents (77.8%) were either neutral or disagreed that the cost of Q Comp could be 
better used for across-the-board raises for teachers. Most (72.3%) were also neutral or 
disagreed that the cost could be better used for other school operating costs. In contrast 
to teachers and administrators, the community awareness survey respondents viewed 
the most significant potential benefit of Q Comp as improved student performance 
(43.1%). The greatest number of respondents indicated improved teacher practice 
(40.6%) as the second most significant potential benefit. 


C. PROGRAMWIDE Q COMP IMPLEMENTATION AND IMPACT 


The evaluation team presents findings from an integrated analysis of survey, focus 
group interview, case study and student performance data by three key areas: the 
nature of Q Comp implementation, the nature of TAP implementation and also impact 
that Q Comp participation has had on state-, district-, school-, teacher- and student- 
level outcomes. 


1. Q Comp implementation 


a. Career ladder/advancement options 
Notably, teachers often appeared to describe components of their salary 
schedule/ performance pay synonymously with the career ladder, and seemed to have 
difficulty distinguishing between the two. Therefore, teacher comments below may be 
out of sync with Q Comp’s intentions for and definition of the career ladder. 


In the implementation survey, teachers and administrators reported interesting 
differences in their perceptions of career ladder existence both prior to and during Q 
Comp (see Figure 6). The majority of teachers believed career ladders were in place 
prior to Q Comp, compared to fewer than half of administrators; conversely, most 
administrators believed Q Comp-specific career ladders are currently established in 
their schools, while far fewer teachers agreed (and 29.4 percent were actually unsure - 
though case study sites were familiar with their career ladders). 
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Figure 6. Teachers and administrators whose schools have a career ladder that: 
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Over half of newer teachers (1 to 3 years experience) were unsure if their school already 
had a career ladder in place before Q Comp (50.9%) or if there is currently a career ladder 
in place associated with Q Comp (58.7%) - compared to only 17.5 percent and 28.8 
percent, respectively, of veteran teachers (20+ years experience). 


Focus group interview teachers from Q Comp schools were generally familiar with their 
career ladder systems, or believed this information to be widely available and easily 
accessible. Teachers from only two case study schools felt the advancement process is 
still unclear and that some roles are not well-defined. However, at most other schools, 
teachers felt once they get someone to help them understand or seek out relevant 
district documents, the formula used to generate the ladder system is fairly 
straightforward (the reference to “formula” may be indicative of teachers’ confusion 
between the career ladder and salary schedule). This is evident in the implementation 
survey, as close to 80 percent of teachers (78.5%) felt they have a thorough 
understanding of the process and requirements by which they advance to higher levels 
on the career ladder, as did three-quarters (75.2%) of administrators. Fifty-four percent 
of veteran teachers (20+ years experience) reported a “very thorough” understanding of 
this process in comparison to 17.6 percent of new teachers (1 to 3 years experience). 
Administrators and teachers generally believed teachers have a clear understanding of 
their role and duties in their career ladder positions (77.9% and 80.5%, respectively), 
and that teachers can effectively handle all the roles expected of them (75.4% and 74.1%, 
respectively). Teachers participating in a Q Comp career ladder seemed to have a better 
understanding about how to share leadership responsibilities with other teachers in their 
career ladder position than with principals (see Figure 7). 
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Figure 7. Teachers’ understanding of how to work together/share leadership 
responsibilities with: 
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@ Principals GB Other teachers in their career ladder positions 


Teachers who participated in focus group interviews indicated the primary means of 
advancement is to take a position of responsibility, though a handful of schools also 
mentioned advancement via classroom observations and professional development; 
nevertheless, the former reportedly advances teachers more quickly and offers higher 
pay (again, teachers may be confusing the career ladder and salary systems). Teachers 
are selected for career ladder positions in various ways. In some schools, selection has 
become a joint decision between the teacher and administrator, whereas in most schools 
there are more formal processes that require applications and interviews. At one 
school, teachers are personally invited by the principal, and teachers could agree to take 
on the extra responsibilities and become part of the administrative leadership team 
(which is open to all teachers); in these positions, teachers work with teacher teams in 
mentoring, coaching, and planning site-based professional growth. Further, Master’s 
degrees are required for advancement in at least two schools. 


Most teachers believed the career ladder system is fair and offers teachers equal 
opportunity (particularly people at younger ages) - though availability of these higher 
positions is limited in some schools, perhaps because some may not be implementing Q 
Comp components as sanctioned by MDE, or are struggling to make such transitions. 
For instance, in one school, advanced positions are held for six to seven years, and it 
takes some time for positions to open once they are filled. Teachers noted there are 
limitations to advancement, though if they are not able to take a particular step, they are 
not necessarily blocked from advancing as there are several layers and directions 
teachers can take to progress up the ladder. Teachers can still be rewarded via the 
professional development they commit to, the observations, and other facets of Q 
Comp. Participants reported that while inherent contention might not be avoidable, 
there is no evidence of “flat out competition,” and the process of accepting teachers into 
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higher positions up the career ladder has been conducted professionally. For instance, 
in one school, candidates interview with multiple people, and anyone can ask to be on 
the interview committee - teachers feel having professional dialogue among several 
people helps make the process more equitable. 


Typically across the schools, as teachers set and meet their personal/ professional goals 
- whether at their own pace or over an academic year - they reportedly move up a step 
on the ladder (again, this is more indicative of salary schedule rather than career 
ladder). Nevertheless, there are reportedly several steps to the advancement process - 
two schools in particular described at least six levels (though it appears that focus 
group interview teachers sometimes interchange career levels and levels associated 
with their alternative salary schedule - i.e., Teacher Levels 1-3, Mentor Teacher, Teacher 
4, Career 1, etc. - which could be an area of further clarification/ edification for 
teachers). Sometimes these levels are limited and cut-and-dry (e.g., choosing among 
only lead teacher, curriculum coordinator, or peer mentor); however, one school in 
particular has two paths teachers may choose after completing the first three levels - 
classroom performance or leadership - which can then branch off into other options 
(e.g., the leadership path veers off to peer leaders, curriculum/ program specialists, 
instructional specialists). Two schools mentioned advancement options/ mentorship 
programs that were available to teachers prior to Q Comp (i.e., curriculum coordinators, 
mentor teachers), which then used Q Comp as a means to pay these teachers for their 
efforts and create more teacher-leader positions. 


This process has changed in several schools since Q Comp’s inception. For instance, 
teachers believed the applications for advanced positions had been amorphous in the 
past, but now that they have set more criteria, there is more respect for the positions 
and the career ladder is easier to follow. 


Generally, teachers enjoy the additional responsibilities in the new advancement 
systems that ladders offer. As one teacher explained, “It’s not like Steps and Lanes; you 
can’t just occupy a seat to go up - you need to do something. You can make a decision 
professionally and personally, and off you go.” Other teachers found it difficult to step 
into these positions due to time constraints, especially those with young families. One 
elaborated of the added responsibilities: 


I hesitate bringing it up - it’s a hard thing to state - for someone who has a lot of 
responsibilities outside of school, it’s a lot to ask of me. I cannot physically do it, 
even though I want to do it. It’s not a discriminatory thing; it is a personal 
choice. But it is a little inflexible in terms of time...It leaves a group of us outside 
of the picture. 


Teachers are sometimes expected to maintain their full-time class loads in addition to 
career ladder responsibilities associated with Q Comp (though some schools do adjust 
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teachers’ workloads). Some described taking on the additional responsibilities and 
worrying about how they will fit it all in later. One administrator added, “The time 
issue, or perception of a time issue, might hold teachers back from trying these - only 
one person in these roles who has tried them said she couldn’t do it. People who have 
stepped forward are invigorated with what they’ve observed. They’ve figured out how 
to manage it.” Administrators in the same district noted there are now 85 teacher 
leaders out of the 382 teachers in the district, compared to only 40 before Q Comp 
arrived. Therefore, despite the difficulties teachers describe, they ultimately seem to be 
pursuing these positions in greater numbers now than they had before Q Comp. 


Teacher pay in the Q Comp charter school included in our site visits was lower on 
average than that of their colleagues in regular public schools. The charter school has 
always had a career ladder system that included lead teachers, curriculum coordinators 
and mentor teachers; Q Comp has allowed the school to compensate their mentor 
teachers. 


Despite the success described by educators in their focus group interviews and surveys, 
both administrators and teachers were either unsure if (39.5% and 36.8%, respectively) 
or did not feel (31.1% and 41.9%) that the teacher career ladder is the major factor for Q 
Comp’s success in their districts. Further, about one-third of both administrators and 
teachers were unsure if the addition of multiple career paths will encourage teachers to 
remain in the profession longer; just under half of administrators (49.2%) felt this will 
have a positive effect on longevity, compared to only 34.8% of teachers. 


About half of non-Q Comp focus group interview teachers were familiar with the career 
ladder aspect of the program, while teachers from four schools were unfamiliar with 
career ladders. Some of the schools described similar facets of the career ladder already 
present in their non-Q Comp systems; for example, AP teachers are paid more in one 
school, and another school informally assigns teachers additional responsibilities (often 
without pay); the latter sees Q Comp as a means for providing teachers with more 
money for their extra work, and as a way to structure formal leadership positions. 
Interestingly, in one school that employs two (non-compensated) career ladder 
positions - content focus coach for literacy and Rt! TOSA (Response to Intervention 
Teacher on Special Assignment) - teachers do not know what the term “career ladder” 
means. 


b. Job-embedded professional development 
Increased focus on student needs and teacher accountability appear to be primary 
consequences of all professional development models described by educators who 
participated in our research. While administrators and teachers sometimes disagreed 
on the execution and impact of their models, all generally agreed that Q Comp has 
added real value to their professional development opportunities. 
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All Q Comp sites by design are to hold regular team meetings. Most case study schools 
described meetings held during the contracted school day, most often during extensions 
added spefically for professional development (before/after school). Three schools 
specifically describe their professional development “clusters” held either weekly or bi- 
monthly - and generally by subject area - while two schools have begun regular team 
meetings in which they collaborate and share ideas. One charter school has designated 
days in which teachers sign up for workshops taught by fellow teachers, which was in 
place prior to Q Comp; the most significant change in this school has been a shift in 
emphasis on topics related to Q Comp goals (i.e., an increase in student achievement in 
math and reading). Across the case study sites, school staff has been tapped more 
frequently to present during professional development sessions, and session topics are 
geared more toward students (i.e., constructed response options) rather than general focus 
areas. Some teachers complete an individual growth plan, which tracks what teachers 
have learned through professional development and how it was applied. Further, a 
handful of schools described the addition or enhancement of mentoring programs. 


In two case study sites, the professional development approach involves a structured 
set of courses required for both new and established teachers. The courses are typically 
taught by teachers who receive training based on a model developed by the American 
Federation of Teachers (AFT), and they have been embedded in the professional 
development system for the past four to five years - before Q Comp implementation. 
Courses for these districts are available during both the summer and the school year, 
spanning several topics - from best practices to content area. Most of the classes offered 
by the programs can be used for credit towards a Master’s degree. Though some 
electives are available, courses are often meant to be interdisciplinary. The models 
typically provide alternatives to the regular course schedule. Teachers can participate 
in study groups organized around a common topic of interest, and they can do 
independent studies. The district limits these activities until a teacher has taken certain 
required courses or after a particular period of time (e.g., once every three years). 


Some teachers from these two sites find the courses exceptionally time consuming - 
teachers are often assigned extra work, such as journal entries, research projects, and 
improvement plans. Further, these teachers noted the redundancy of many best 
practice courses, and that the focus on interdisciplinary strategies makes less room for 
some subject areas (i.e., science, foreign languages). While some find this repetitive, 
others find this a convenient refresher and a way to equip educators with aligned 
strategies. However, teachers also noted some inflexibility in the structure, as they are 
encouraged to provide classroom instruction in line with their professional 
development course format. This format reportedly becomes repetitive in the 
classroom - though while some find it difficult to meet both the professional 
development requirements and the needs of their students, others find ways to adjust 
and take what they need from the courses. 
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According to the implementation survey, topics most often addressed in professional 
development across Q Comp schools include data-based decision making, student 
assessments, instructional techniques, and differentiated learning (see Table 8). 
Administrators generally felt these topics should be addressed by the professional 
development activities, while teachers believed increased emphasis should be placed on 
content area/ grade level, with perhaps less focus on data. 


Table 7. Professional development topics (Admin n =180; Teacher n=1872) 


Actually covered | Should be covered 
Response Administrators Teachers Administrators | Teachers 

Data-based decision 

making 90.0% 72.2% 93.9% 60.1% 
Student assessments 87.2% 75.9% 93.9% 66.1% 
Instructional techniques 84.4% 63.8% 90.0% 78.3% 
Differentiated learning 80.6% 71.0% 92.2% 73.6% 
Standards-based lessons 66.7% 51.9% 81.7% 58.5% 
The content area/grade 

level teachers teach 64.4% 52.2% 67.8% 70.0% 
Small group instruction 51.1% 38.3% 67.2% 54.0% 
Classroom management 49.4% 41.8% 62.8% 57.9% 


Note: Percentages exceed 100 due to respondents being able to choose more than one answer; bolded 
numbers represent the four highest percentages for each column. 


Both administrators and teachers reported that more often than not, teachers within the 
school are tapped to share insights and ideas for professional development content - 
though teachers were more inclined than administrators to report that outside experts 
are the primary professional development approach, and that the two approaches are 
not typically combined (see Figure 4; this is not necessarily reflective of what teachers 
want in schools, but is rather what they already observe). Further, administrators and 
teachers believe professional learning communities for teachers are present in their 
school to either a moderate or great extent, though administrators were slightly more 
confident of this (87.7%) than teachers (74.9%). 
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Figure 8. Primary approach to professional development in schools: 


Teachers within the 74.7% 
school are tapped 67.8% 


Outside experts are 19.3% 
brought in 31.3% 


The two approaches are 70.6% 
combined 54.9% 
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There appears to be, according to the statewide implementation survey, a sizeable 
disconnect between administrators and teachers regarding the alignment of 
professional development with needs and goals, as well as its development and 
execution. The overwhelming majority of administrators generally believe their job- 
embedded professional development aligns with their school/ district improvement 
plan, aligns with both teacher and student needs, takes place during the teacher 
contract day, and is developed/controlled by the teachers (see Figure 5). Teachers, 
while typically in agreement with administrators on most items regarding professional 
development, were far more likely to feel uncertain or to disagree with these statements 
altogether. 


Figure 9. Administrators and Teachers who feel job-embedded professional 
development in their school/district typically: 


Aligns with our school or 
district improvement plan 
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Aligns with our teachers' 57.4% 
Is developed/controlled by 45.9% 
the teachers in our school 78.3% 
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Most teachers believe that under Q Comp, the professional development has become 
much more valuable and relevant to teachers; it finally seems to have a purpose and can 
be applied immediately to the classroom. One teacher noted that, compared to 
professional development over the past 25 years, the training and opportunities have 
improved so much that the comparison is not even close. Q Comp seems to have 
embedded a sense of personal accountability, as teachers are expected to take what they 
learn back to their classrooms and follow up with the results. In fact, in at least two 
schools, classroom application of professional development is integral to the Q Comp 
evaluations. One teacher, in line with the general consensus, explains: 


This is professional development with a purpose - you're so disconnected after 
college. There are specified courses, but you can take ones that make sense to 
you, and you can change things in your classroom. What have you done, and 
what data you can take on affects your [evaluation] score. You do hands-on 
manipulating, and then report back to talk about how it went. You have to show 
you demonstrated research, and that you just try - you know when things are 
better than they were before, or what students respond to. 


Slightly greater than half of administrators (61.7%) reported in the implementation 
survey that job-embedded professional development is the major factor for Q Comp’s 
success in their district, compared to 36.7 percent of teachers, despite the improvements 
they reported in their focus group interviews. About one-third of teachers (32.5%) 
remained neutral, along with 22.8 percent of administrators. 


Most teachers who participated in non-Q Comp focus group interviews were familiar 
with the concept of job-embedded professional development. About half the sites 
acknowledge this model is not common in their own schools. For many, professional 
development is currently provided through a combination of district offerings and 
outside workshops, which is reportedly not sufficiently flexible to meet the variety of 
teacher needs. Rather, many teachers, primarily by grade level, gather informally to 
share information and improve their teaching practice. At least four sites have weekly 
or bi-monthly PLCs covering a variety of topics. Three schools recalled limited 
mentoring programs, one of which only lasts a year for new teachers; two others 
mentioned informal mentoring in which every teacher takes someone under their wing. 
One teacher says, “The district level has dropped the ball in terms of leadership so we 
have to do it ourselves.” Teachers were clear that there were many knowledgeable 
people right in their building who could provide them with quality development in 
various areas. One administrator (whose teachers are particularly unknowledgeable 
and skeptical of Q Comp) confirms this, and remarks, “They are already going down 
that [Q Comp] road and just don’t know it.” Administrators described financial 
constraints with professional development. A few schools were interested in looking at 
Q Comp to improve their models and teacher salaries, as they feel it is a legitimate 
means of providing more variety and accountability to their professional development. 
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c. Teacher observations/evaluations 
In Q Comp sites, all teachers are evaluated and observed, regardless of experience level. 
They are observed by a variety of people, including principals, vice principals, teacher 
leaders, coaches, mentors, and/or peers. Conferences occur before and after 
observations where the teachers’ own ratings are also considered. Observation training 
is provided and a specific protocol is used so the observer can be as objective as 
possible. A case study participant indicated that after joining Q Comp, the observation 
process has been “fine tuned” to focus on instruction and school goals and be more 
specific in dialogue about “school achievement.” 


According to a survey, both administrators and teachers indicated that teachers are 
typically observed or evaluated in the classroom once a semester (51.4% and 71.3%, 
respectively) or once a month (32.8% and 16.6%, respectively). Given the right 
messaging, Q Comp teachers can be made to feel at ease with observations, knowing 
there will be several during the year rather than only one, and the purpose of the 
observation is to enhance, not criticize instruction. A Q Comp case study teacher 
commented on evaluations: 


It’s fine because we have a safe environment. There is trust. We get helpful 
comments, support and assistance. We're not being judged. The administrators, 
coaches and mentors are in our classrooms so much that we don’t get worried. 
They take into consideration what they have seen all year, not just on one 
particular day. 


In a survey, more than 70 percent of both administrators and teachers felt that teachers’ 
performance is evaluated fairly and accurately under Q Comp, and that teachers are 
evaluated based upon defined teaching standards (see Table 9). More than three- 
quarters of teachers (78.3%) also agreed with the statement, “I use results and feedback 
from classroom observations to improve my teaching and focus.” More administrators 
than teachers agreed with statements such as “teacher observations and evaluations 
under Q Comp measure important aspects of teachers’ performance” and “teachers are 
evaluated based on a clear rubric that scores Q Comp’s teaching standards.” 


Table 8. Teachers and administrators who agree or strongly agree with 
statements about classroom observations and teacher evaluations 
Statement Teachers Administrators 
My performance is (Teachers are) evaluated fairly 
and accurately. 81.8% 73.6% 
| use results and feedback from the classroom 
observations to improve my teaching and focus 


where | need to grow professionally. 78.3% n/a 
(My) teaching is evaluated based on defined 

teaching standards. 76.8% 84.1% 
Those in the Q Comp system who evaluate my 

(teachers’) performance against Q Comp goals are 70.3% 68.2% 
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impartial. 

My teaching is (Teachers are) evaluated based ona 
clear rubric that scores Q Comp's teaching 
standards. 62.7% 78.5% 
Teacher observations and evaluations under Q 
Comp measure important aspects of my (teachers’) 


teaching performance. 63.7% 83.7% 
Q Comp accurately distinguishes effective teachers 
from ineffective teachers at my school. 18.4% 47.2% 


In contrast, at non-Q Comp sites, non-tenured teachers are typically the primary focus 
for any administrators conducting observations. Unlike Q Comp, usually the principals 
or vice principals are the only individuals who observe teachers. However, some 
teachers are being taught to conduct “learning walks” in which they collect evidence, 
then, do “wanderings.” In non-Q Comp sites, teachers particularly feel uncomfortable 
with observations for a variety of reasons. In lieu of observations, teachers work on 
goal setting (at designated times of the year), but a consistent reflective practice is not 
apparent. When one non-Q Comp administrator was asked how teachers might react to 
having Q Comp at the school, he shared: 


It brings teachers to a level of professionalism they’ve never had before. They get 
rewards for successes. The downsides are getting used to being evaluated and 
the transition to rubrics. I think after really pitching it, about half of the teachers 
at this school would welcome Q Comp. 


d. Performance pay 
Schools who have implemented Q Comp tend to view it as an integrated program 
involving each of the five components, not just performance pay. Indeed, the program 
does seem to be well-integrated, with all five elements reinforcing one another. 
However, non-Q Comp schools who have considered implementing Q Comp tend to 
view it as a vehicle for providing more money to teachers. They also express doubts 
that funding for the program will be permanent, and therefore see little reason to 
implement the program (i.e., if it’s only about the money, and the money’s not 
guaranteed to be there, they see no reason to do it). Convincing non-Q Comp schools of 
the value of the non-compensation aspects of Q Comp will be critical if more schools are 
to join the program. 


The design of the performance pay components in Q Comp schools appears to be well- 
balanced in terms of the types of performance incorporated. In most schools, teachers 
are paid more for higher performance evaluations, additional responsibilities on the 
career ladder, higher test scores, and professional development activities. All 
applications have schoolwide goals and all Q Comp participants (districts, charters and 
school sites) are required to send updated schoolwide goals to MDE annually. Schools 
have had to provide a detailed definition in their application of how progress will be 
defined and measured. 
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In the Q Comp survey and in the focus group interviews, teachers generally indicated 
that they are happy with the performance-based pay components of Q Comp and 
believe they are fair (see Figure 10). In the survey, there were some differences in views 
based on how much experience a teacher has, but the differences were small. Teachers 
in one focus group interview confirmed they felt being compensated for effort (as in the 
Q Comp system) is fairer than being compensated for experience. 


Figure 10. Attitudes of teachers/administrators regarding Q Comp’s 
compensation system 
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Feelings were mixed among teachers regarding whether the use of student test results 
in the compensation system was fair. In some schools, teachers did not show anxiety 
about test results being used. However, in one school there was some concern about 
using MCA-II as the primary measure of student results, given that it does not account 
for variations in the ability of students in a classroom from year to year. Some were 
uncomfortable with the idea that performance pay would be based on the results of a 
single test administered on a single day. In one school, there were only minimal 
concerns about the use of test results in the compensation system in the first year of Q 
Comp, when the schoolwide goals were achieved. However, in the second year, the 
school did not hit their schoolwide goal (and therefore teachers did not receive 
additional compensation), and teachers began to express more reservations about the 
use of the MCA-II results. Administrators generally were more supportive of using 
student test results in the compensation system. 


Almost all teachers felt that it was appropriate to pay additional compensation to 
teachers who had additional responsibilities associated with a career ladder (83.9 
percent of administrators and 80.0 percent of teachers agreed or strongly agreed with 
this statement). However, at one school, some teachers commented that this was unfair 
because teacher leaders had a reduced teaching load and were making more money. 
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The additional pay provided by Q Comp was seen by a number of teachers as 
compensation for extra work, as opposed to purely for performance. Teachers haven't 
necessarily bought into the idea that paying for performance as opposed to effort (or 
professional development activities) is appropriate. In the Q Comp survey, only a 
limited number of teachers thought that teacher performance and student performance 
on standardized tests should be important or very important in determining base pay 
and supplemental pay (see Tables 10 and 11 in next section). Potentially feeding into 
this notion is the fact that the design of the performance-based pay component relies 
heavily on effort-based measures in some schools, including paying teachers for 
attending specified professional development courses. 


Feelings were somewhat mixed among teachers about whether the additional pay 
provided by Q Comp made up for the additional work required. In some schools, 
teachers believed that Q Comp rewards teachers fairly for the work that is required of 
them, while teachers in other schools feel the pay is minimal, relative to the additional 
hours of time. In the Q Comp survey, 45.8 percent of teachers agreed or strongly agreed 
that the potential monetary amount teachers can receive is appropriate given what they 
are being asked to do in Q Comp, while 34.6 percent of teachers disagreed or strongly 
disagreed. 


Teachers, mentors, and coaches share a common concern that the financial gains behind 
Q Comp are not the driving motivator. Many teachers in Q Comp schools expressed in 
the focus group interviews they “don’t need money to do what's good for students.” 
One participant commented, “For the amount of time and work, the money is quite 
little. We were the ones who said money wasn’t important. We don’t want to help 
students for money; it’s intrinsic in our jobs that we want to help students.” Even 
though the additional pay was greatly appreciated by teachers, this sentiment was 
common. Teachers seemed to feel it would be inappropriate for someone in their 
profession to be motivated by money. Indeed, in the Q Comp survey, only 41.6 percent 
of teachers agreed or strongly agreed that Q Comp encouraged them to work harder 
than in prior years to receive more compensation. 


Q Comp programs are, by and large, well understood by teachers (in the Q Comp 
survey 76.0 percent of teachers said they thoroughly or very thoroughly understood the 
criteria by which they earn greater compensation). However, many teachers expressed 
in the focus group interviews that the new systems were difficult to understand 
initially, and that the involvement of the administration, teacher leaders, or aQ Comp 
coordinator was critical in helping teachers to understand how the system worked. 


As a corollary to this observation, administrative focus group interviews indicated that 
the implementation, communication, and administration of the compensation portions 
of Q Comp can be quite time consuming. Given this, smaller districts or charter schools 
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may not have sufficient resources to develop and administer a Q Comp pay system. 
Since teachers expressed that the pay components of Q Comp need to be carefully 
implemented to avoid any negative outcomes, this could be a potential area of concern. 


e. Alternative salary schedule 
Because in many Q Comp schools the alternative salary schedule is tied so closely to the 
performance-pay component of Q Comp, many of the same observations apply. 
However, in discussions with non-Q Comp schools, there was confusion as to exactly 
what is required for the alternative salary schedules. There was also hesitancy and 
resistance to change the current Steps and Lanes schedule, and uncertainty about what 
a suitable replacement would look like. Even in Q Comp schools, the traditional Steps 
and Lanes system still remains prominent in teachers’ minds - teachers still believe, 
according to the implementation survey, that years of experience and highest academic 
degree earned should play the primary role in determining base pay and a significant 
role in determining supplemental pay (see Tables 10 and 11). However, the survey also 
provides evidence that less experienced teachers are more open to performance-based 
salary schedules (and supplemental pay) than teachers with significant years of total 
teaching experience, particularly when performance is determined by principal 


observations/ evaluations. 


Table 9. Important factors in determining teachers' base pay 


Years Teacher Experience 


Administrators 


67.2% 


All 
91.7% 


Teachers 
1-3 Years 
Experience 


75.2% 


>20 years 
Experience 


96.5% 


Highest Academic Degree Earned 


74.5% 


90.5% 


84.4% 


91.2% 


Teacher performance, as 
determined by principal 
evaluations, observations, 
teacher portfolios, etc. 


70.5% 


51.4% 


62.4% 


46.6% 


Student performance on 
standardized tests at the school 
level 


33.9% 


14.2% 


17.4% 


15.8% 


Table 10. Important factors in determining teachers' supplemental pay 


Years Teacher Experience 


Administrators 
16.1% 


All 
43.2% 


Teachers 
1-3 Years 
Experience 


23.8% 


>20 years 
Experience 
55.8% 


Highest Academic Degree Earned 
Teacher performance, as 
determined by principal 
evaluations, observations, teacher 
portfolios, etc. 


25.0% 


76.7% 


47.5% 


54.8% 


40.3% 


70.6% 


54.8% 


46.9% 


Student performance on 
standardized tests at the school 
level 


53.9% 


24.4% 


36.7% 


22.0% 
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The difficulty in transitioning away from the Steps and Lanes schedule was confirmed 
in the focus group interviews of one Q Comp school; these teachers reported the change 
to anew salary schedule was enormously difficult, but the fact that there was more 
money involved made the transition easier. On the other hand, a charter Q Comp 
school indicated that since they did not have a Steps and Lanes system before Q Comp, 
the salary system is now much more transparent and better understood than it was 
prior to Q Comp. 


Generally, Q Comp participants reportedly advance in salary by meeting 
requirements/ goals from reviews, being observed, taking a position of responsibility 
(career ladder), demonstrating improvements in student performance (schoolwide test 
results), and spending time in professional development activities (i.e., PLCs). One 
school notes it can take several years to change salary positions. 


In two Q Comp schools, an alternative pay schedule is not yet fully developed. The 
alternative pay schedule is required, but some districts are under timelines to create one 
so the alternative pay schedule is still in process or being developed. According to 
teacher perception, each individual teacher has the choice to move to the new salary 
schedule (as initially instituted by a districtwide vote) or not participate and stay on the 
traditional salary schedule (Steps and Lanes). Those who participate are compensated 
according to criteria mentioned above (i.e., observation results, time spent in 
professional development, test scores). MDE has clarified that one district has been 
given permission for a transitional timeline during which teachers can choose when 
they transition due to the drastic differences in the two schedules and the district’s four- 
year MOU. 


Two sites coexisting in a K-12 building created a new schedule that still essentially has 
“Steps and Lanes,” but there are fewer steps for experience and more lanes for 
education and credits earned. Teachers are no longer paid strictly for years of 
experience, as was the case prior to Q Comp - but rather for proficiency in criteria 
mentioned earlier (primarily observation results). 


One small charter school reportedly had no salary schedule or union in the school 
before or after Q Comp. Teacher pay was lower on average than in regular public 
schools. Before Q Comp, teachers received approximately a 1% raise each year; they 
also received extra compensation for the additional roles they assumed in the school. 
This school applied for Q Comp in the first year it became available in order to obtain 
more money for teachers. This latter motivation was common among schools; in fact, 
another Q Comp school utilized this in their recruitment. Ata recruitment fair, the 
school hung a banner that included the much higher starting pay (thanks to Q Comp 
funds), and they reportedly had a line out the door with interested teachers. 
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One district in particular utilized a “strategic transition” system, as it made more sense 
for some teachers to wait a while before moving into the Q Comp system. As there are 
costs associated with moving teachers into Q Comp, there was some indication in the 
focus group interviews that the district may need to formally limit transitions by 
teachers still on Steps and Lanes in the future. 


Another school experienced confusion in that the district’s procedures regarding both 
teachers who had come from other districts and those who were already long-time 
teachers before Q Comp were unclear. Prior service credit was described as “tricky.” 
The teachers on “justified advancement” or “catch up” programs under Q Comp 
seemed to understand these salary adjustments, but other teachers who were not 
directly involved were not as familiar. One teacher felt that under Q Comp, the salary 
structure was not equitable to newer teachers until they received a Master’s degree - in 
fact, both sites in this district report they are expected to obtain a Master’s before 
advancing in salary; the difference in salary could be as much as $30,000 later in their 
career, according to one teacher. These procedures, however, were clearer for the 
district’s other site; teachers who had been with the school prior to Q Comp were given 
the option to either stay with Steps and Lanes or move into the new salary schedule - 
new teachers coming in after Q Comp move right into the new system. 


2. TAP implementation 


The Teacher Advancement Program (TAP)" is being implemented in two districts in 
Minnesota, totaling 15 participating schools for the current school year. TAP 
requirements are more intensive than Q Comp. TAP has four components (no 
alternative salary schedule) and Q Comp has five components. However, the 
professional development for TAP schools is particularly rigorous. When asked what 
TAP was, these teachers most often commented on the professional development 
meetings to enhance and unify instruction. One of the TAP districts has a district level 
lead TAP mentor. This position has been instrumental in supporting TAP participants 
and pursuing additional participants. Each of the TAP schools in this district are visited 
on an ongoing basis by the lead TAP mentor. The TAP schools are each equipped with 
a TAP mentor(s) and a TAP coach(es). 


TAP schools have weekly meetings for administrative leaders and weekly or bi-weekly 
teacher training meetings (called clusters). A teacher described, “It (TAP) is building 
relationships across the whole school. We’re all equals, even the administration -we’re 
all struggling with the same things. It’s not threatening.” While teachers mostly see 
TAP as a means of increased collaboration, additional training, improved instruction 
and reflection, administrators at case study sites additionally viewed TAP as a resource 
for funding and staff. A TAP administrator explained, “It’s been a good experience and 


1 See glossary in Appendix 1. 
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good segue into financial support that we needed for professional development. There 
will be barriers to overcome unless you have financial backing. I would be crushed if 
TAP wasn’t voted in.” 


TAP principals, assistant principals, coaches, mentors, and teachers contributed to two 
of the case study sites. The participants disclosed numerous benefits of TAP. For 
example, a TAP mentor described: 


The principals are actively involved as educational leaders. The school plan 
works and is focused and valuable. Each year we are more fine-tuned in our 
strategies. We are creating rubrics for writing assignments now. Teachers are 
now ina high level of monitoring their work. We are very pleased. Students are 
doing better on MCA-IIs. 


Outside MDE, there are mixed understandings of the distinctions and similarities 
between TAP and Q Comp. This was particularly evident when an administrator at a 
TAP school indicated that the teachers wouldn’t know what was meant by “Q Comp” 
during the case study focus group interviews. It was suggested to substitute “TAP” in 
the protocols whenever “Q Comp” appeared. When non-TAP (Q Comp) teachers were 
asked if they would like to participate in TAP (Q Comp) in the future or what benefits 
and drawbacks do they envision if they were to participate, multiple case study 
teachers (usually at least half) said they had just heard about TAP (Q Comp) that day or 
they didn’t know enough to know if they would like to get involved. 


3. Q Comp impact 


Given the relatively limited number of schools participating in Q Comp (see Table 12), 
the impact of the program on a statewide level is difficult to extrapolate. Ata 
minimum, Minnesota has attracted national attention with the program, with numerous 
media and journal articles being written about Q Comp. Also, given the effects at the 
school and teacher level, Minnesota teachers appear to be more supported in their role 
of improving students' educational achievement since Q Comp’s inception, though 
these trends cannot be linked statistically. 


Table 11. Statewide Participation in Q Comp _ 
Number Participating in Q % Participating in Q 
Comp Comp 
Districts (nN=336) 39 11.6% 


Charter Schools (n=143) 21 14.7% 


Since the evaluation has been very brief and preliminary, the evaluation team cannot 
comment on whether Q Comp makes the teaching profession in general more attractive 
to potential teachers (i.e., Does the presence of a program like Q Comp encourage more 
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individuals to join the teaching profession?). Opinions differ regarding the 
attractiveness of Q Comp to new teachers. In the Q Comp Survey, 19.4 percent of 
teachers and 51.1 percent of administrators think Q Comp makes a district/school more 
attractive to teachers they want to hire. While the evaluation does not have a “before” 
and “after” view of the public’s opinions of performance-based pay for teachers, the 
stakeholder survey did indicate that 69.2 percent of respondents believe it is 
appropriate to pay teachers based on their classroom performance and 71.2 percent of 
responders would support implementing Q Comp at their local school. 


a. District-level 
The greatest impact of Q Comp can be felt at the school, teacher, and student levels. 
However, there are several points to be made at the district level. Some Q Comp 
administrators cited that there has been increased consistency from school to school 
within a Q Comp district in the way that expectations for students are set and in the 
teaching strategies that are being used. Additionally, some districts described the 
professional development activities associated with Q Comp that bring teachers from 
different schools together and enable them to share ideas and best practices. 


There were also several potentially negative effects of Q Comp implementation 
observed at the district level. One smaller district reported there was less 
communication between schools within a district, given the focus on team building 
within each school. In addition, Q Comp may make it slightly more difficult for 
teachers to transition from district to district, given the differences in the design of the Q 
Comp systems. It might be difficult for a teacher to learn a new professional 
development system, career ladder, and compensation system when switching districts. 


b. School-level 
The impact of Q Comp at the school level has, overall, been quite positive. There have 
been substantial changes to school culture, instructional practices, schoolwide 
communication, focus, and management structure/ practices of Q Comp schools. 


(1) School culture 


Several significant changes to school culture were revealed in the focus group interview 
discussions at case study sites. The primary change described was that Q Comp schools 
have become more collaborative environments, with significantly more sharing 
amongst teachers. Time set aside to collaborate with peers was one of the most valued, 
beneficial aspects of the program. Even though the various Q Comp sites had different 
amounts of collaborative time during the school week, all Q Comp sites observed this 
change (greater collaboration with staff was the top benefit to Q Comp reported by both 
teachers and administrators in the Q Comp survey). In particular, teachers’ increased 
collaboration involved more sharing of instructional practices, ideas, and strategies. 
Teachers in Q Comp schools also have a chance to see how others give lessons and to 
learn from those experiences. They were pleased to have opportunities to see what 


Hezel Associates, LLC 38 


Q Comp Summative Evaluation Final Report for Public Release 


other teachers were doing in the classroom as well as to share what they themselves 
were doing. As one teacher observed in a focus group interview, “At other schools we 
keep our good ideas to ourselves. Here, we share our teaching styles, rather than keep 
our secrets.” Teachers also feel less isolated in their classrooms, with more interaction 
with their colleagues. There is more of a team spirit, with teachers seeing efforts and 
struggles as buildingwide, not only at the individual level (e.g., “Everyone takes 
responsibility for student growth -it’s ‘all’ of our students, not just “your’ students”). 


Related to the increased collaboration was a stronger sense of community in the school 
(e.g., “more tight-knit”). Teachers described feeling more collegiality with other 
teachers and a sense that “we're in this together.” A teacher noted that the program 
gives teachers an opportunity to get to know others beyond their own grade level, and 
to “build relationships across the whole school.” Of particular note was that newer 
teachers have a venue to get to know and learn from their more experienced colleagues. 


One of the concerns regarding the Q Comp program is that it would cause increased 
competition between teachers. By and large, this problem was not observed at the case 
study sites. When probed, teachers in one focus group interview indicated that there 
might be a small amount of additional competitiveness, but most teachers indicated that 
this was not an issue, and that increased collaboration was by far more common. The Q 
Comp survey confirmed this finding, as 79.4 percent of administrators and 72.3 percent 
of teachers disagreed or strongly disagreed with the statement that teachers have 
become more competitive than cooperative with Q Comp. 


Another change to school culture discussed in the focus group interviews and observed 
in the schools was that the schools had a more positive environment. Teachers were 
using their time together to discuss the practice of teaching rather than to complain 
about the school or the students. The environment was also more supportive, which is 
evidenced in the implementation survey - 84.6 percent of responding teachers felt 
supported by other teachers in their grade. 


(2) Instructional Practices 


All of the case study sites believe that instructional practices have improved under Q 
Comp. Administrators and teachers observed: 


e More consistency and cohesion in how teachers teach, and therefore in the 
student experience from class to class. In a focus group interview, teachers in 
one school observed: “Teachers use similar terminology, strategies and 
approaches that it is unified as students go to different classrooms. Students 
know what is expected of them and what skills they are working on. The rooms 
even have similar displays to promote consistency.” In another school: “I don’t 
have to start over again in getting new students each year because they have 
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heard certain terms before because we all learn and use them. The students know 
the routines.” 

e A willingness to revise the curriculum when necessary, more deliberate 
construction of lessons, increased use of proven instructional strategies, and 
better instructional delivery (e.g., “better questioning and discussion”) 

e Teachers use student performance data more often to improve classroom 
instruction - 87.3 percent of administrators and 69.0 percent of teachers agreed or 
strongly agreed with this in the Q Comp survey. Teachers were also observed to 
be more focused on student test results as an indicator of success. 

e Teachers set clearer, more effective instructional goals - 82.2 percent of 
administrators and 61.8 percent of teachers agreed or strongly agreed with this in 
the Q Comp survey 

e Asa result of Q Comp, teachers encourage students to keep trying even when 
the work is challenging - 68.9 percent of administrators and 65.6 percent of 
teachers agreed or strongly agreed with this in the Q Comp survey 

e Increased focus on student needs on differentiation (e.g., “we're adjusting our 
teaching based on students’ needs”) 


(3) Schoolwide communication 


Almost all of the case study sites mentioned improved communication as a significant 
impact of Q Comp. This was primarily discussed as it related to Q Comp giving 
administrators and teachers a common language/vocabulary to use (e.g., “there is a 
schoolwide continuum in which we talk using the same language”). 


Communication regarding students also improved, given the increased level of contact 
between teachers. Teacher meetings allow teachers to “know what specific students are 
struggling with” and then adapt accordingly. 


Some sites also mentioned a better flow of information from the administration to the 
teachers, because teacher leaders were a part of the school leadership team and also 
worked closely with non-leader teachers. 


The only negative in terms of communication that was observed was at a smaller 
district, where the focus was on improving relationships and teaming by grade/section 
(e.g., high school, middle school, elementary), and as a result, communication with 
teachers between sections suffered. 


(4) Unified focus on and monitoring of schoolwide goals 


Another school-level impact of Q Comp is that it provides a unifying focus and 
framework for participating schools and their schoolwide goals. Schools reported 
having common performance goals that provided clear expectations for both teachers 
and students, and this streamlining reportedly makes monitoring goal progress clearer 
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for all parties. One school reported that its schoolwide performance goals and focus 
were on student achievement in reading and math - this focus is seen in teachers’ 
professional improvement plans, monitored in teacher observation rubrics and, 
ultimately, rewarded with performance pay. Another school reported that they had a 
unified focus on reading, with their teachers “working more cohesively toward a single 
goal.” These common goals, emphasized by Q Comp, provided a unity of purpose 
stemming from a collaborative, student-centered focus. 


One district utilized Q Comp as a jumpstart to setting concrete building goals and 
sticking with them. The program gave peer leaders the opportunity to work with other 
building leaders in settting these goals, and staff members meet regularly to discuss 
how they are meeting those goals. 


The other advantage of Q Comp in this area is that it provides a framework into which 
other initiatives can be folded. Q Comp provides a holistic structure that can tie in 
other initiatives, causing schools to feel more organized and teachers to feel less like 
there are too many things going on at once. 


(5) Leadership structure & practices 


The administration at two case study sites described the role of the principal and how 
teachers were managed, given that teacher leaders reduce a principal’s span of control. 
One site described an overall shift from administrative decision-making to teacher 
decision-making. Another described the benefits of having teacher leaders who are 
closer to the teachers than the principal may be - a teacher reported, “Sometimes, it’s 
hard to approach administrators and suggest different ways of trying something, so I 
like having coordinators who are teachers, who wear a lot of hats and are open, easy to 
talk to, willing to try new things.” 


c. Teacher-level 
Under Q Comp, teacher level impacts included changes to teacher attitudes, 
professional development, workload, and compensation levels. 


(1) Teacher attitudes 


One of the most common changes in teacher attitudes was a renewed focus on their 
own development and teaching practices. One teacher remarked that Q Comp was 
“reinvigorating teachers across the board.” Another remarked, “How do you not 
become an old teacher that does the same things? Q Comp doesn’t let me get into a 
rut.” One school noted that initially, teachers were quite set in their ways and less 
flexible to change their teaching practices; however, after three years under Q Comp, 
teachers are more flexible and willing to learn from one another. 


Teachers were seen as being more reflective in their teaching practices (e.g., reflecting 
on lesson plans to improve student learning). In one school, teachers indicated that 
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journals and meetings force them to look at what they are doing and what would be 
helpful to change in their instructional delivery. They want feedback after observations 
and ideas from other teachers during meetings, and strive to do well and are more 
aware and conscientious of their teaching. The Q Comp survey confirmed this finding, 
as it indicated that teachers are better able to identify their strengths and weaknesses 
and use the feedback from their classroom observations to improve their teaching (see 
Figure 11). The survey data also showed that less experienced teachers tend to be more 
reflective than more experienced teachers. 


Figure 11. Teacher/Administrator attitudes on outcomes as a result of classroom 
observations 
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An additional change in attitudes worth noting relates to observations and feedback. 
Teachers traditionally have been isolated in their classrooms, with visitors to their class 
providing a source of anxiety. However, in Q Comp schools, teachers were more 
comfortable with administrators and other teachers observing them and providing 
feedback. One of the reasons that teachers were able to be comfortable with this is that 
they felt these observations were “non-judgmental and non-threatening.” Observations 
were seen as an opportunity to learn and to share their expertise with one another - one 
teacher said, “We're ok with admitting our faults” now. In one school, however, a 
teacher noted that there had been some instances of teacher observations where 
“friends evaluating friends” caused tension. 


(2) Professional development 


Under Q Comp, teachers have access to more unified and focused professional 
development. One teacher noted that they no longer go to isolated sessions where they 
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do not share or apply what they have learned. Rather, the techniques are implemented 
and evaluated and weekly meetings allow for ongoing discussion and a chance for 
teachers to spread their expertise. As another teacher noted, they had professional 
development that actually “has a purpose.” The Q Comp survey confirmed this 
finding, as 95.0 percent of administrators and 57.4 percent of teachers believe job- 
embedded professional development at their school aligned with teachers’ needs (and 
91.0 percent and 65.6 percent, respectively, for aligning with students’ needs). 


(3) Workload/time requirements 


Nearly all case study schools noted an increased workload for teachers due to Q Comp 
- more meetings, paperwork, and requirements that tax available time - and often 
longer schools days as a result of extended contracted time for activities like PLC 
meetings. Some felt that these increased requirements were particularly onerous on 
new teachers, who may be spending more time just getting up to speed with their 
teaching. The reaction to this increased workload was mixed across the case study 
schools. At one site, teachers did not complain about the increased workload or 
meetings because they felt that they were being compensated for it and that it was 
beneficial to themselves and their students. Another said that the increase in workload 
was initially quite difficult, but teachers have now adjusted and learned to handle it. In 
another school, teachers were initially comfortable with the additional workload, but in 
a particular year when they did not hit their schoolwide goal (and therefore did not 
receive additional compensation), teachers began to complain about the work involved 
(meeting time, writing professional improvement plans, and writing district and team 
goals). One teacher said that, “new teachers are staying away from Q Comp because of 
the extra work and extra stress.” 


However, teachers in Q Comp schools also spoke of less “wasted time.” Meetings were 
more efficient and relevant, as were professional development sessions. While there 
was some benefit in terms of time commitments with Q Comp, overall, it is fairly clear 
that the system requires more time of teachers. The degree of comfort with this 
additional time required appears to be based on whether the teachers believe they are 
being fairly compensated and the investment of time is worthwhile. 


(4) Increased compensation 


Though many teachers indicated that they did not need additional money to be 
motivated to do what was right for their students, they also believed that additional 
compensation was only fair, given the increased time commitment required. Teachers 
also felt that basing this increased compensation on effort, as opposed to experience, 
was fair. 
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d. Student-level 
In this evaluation, evaluators attempted to reveal the set of Q Comp implementation 
strategies that, in combination, best predict or produce success in student achievement. 
There is a wide variety of Q Comp implementation strategies that districts used, and 
this evaluation identified as many as possible in the teacher and administrator surveys. 
Therefore, responses to these surveys are the predictor variables used in the regression 
equations. MCA-II scores are the dependent variable as they represent student 
achievement. 


Results of all three sets of regression procedures are provided in Appendix 3. Each 
regression equation provides an R coefficient and an R Squared for each independent 
variable. The first set of regression equations, using all 232 predictor variables from the 
surveys, was done with 2008 MCA-II reading and math scores in grades 3, 8 and 10/11. 
These regressions identified 11 variables that produced the best set of predictors for 
student academic success as a result of Q Comp implementation. These variables are: 


Ala - Admin. Survey: Teachers consider Q Comp to be successful so far in our 
district. 

A2a - Admin. Survey: Level of input the following groups had in developing 
their school’s/ district’s Q Comp plan - Individual teachers. 

A8c - Admin. Survey: How could the evaluation process be improved? - 
Evaluations conducted by better trained/ more experienced teachers. 

AQa - Admin. Survey: Degree you agree to the following statements: Those in 
the Q Comp system who evaluate teacher’s performance against Q Comp 
goals are impartial. 

A16b_b - Admin. Survey: Indicate to what degree you agree with the following 
statement: Teachers have a clear understanding of their role and duties in 
their career ladder positions. 

A19a - Admin. Survey: Job embedded professional development in your 
school/ district typically takes place during the teacher contract day. 

T6a - Teacher Survey: Who is responsible for conducting Q Comp teacher 
evaluations/ observations? Principal 

T6b - Teacher Survey: Who is responsible for conducting Q Comp teacher 
evaluations/ observations? Principal/ Vice Principal 

T16b_a - Teacher Survey: To What degree do you agree with the following 
statements: With the addition of multiple career paths, I will remain in the 
teaching profession longer. 

T21e - Teacher Survey: What topics do the professional development activities 
and discussions in our school address? Standards-based lessons 

T21f - Teacher Survey: What topics do the professional development activities 
and discussions in our school address? Student assessments 
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The second set of regression equations was conducted to determine if additional 
analyses from 2006 and 2007 MCA-II scores would change the variable combinations 
that best predict student achievement. Again, all 232 variables were entered into 
regression equations for both subjects and for three levels (grades 3, 8 and 10/11) for 
2006 and 2007, just like they were for 2008. The results indicated that nine out of the 11 
original variables from the 2008 analyses were also included in at least one regression 
equation in these additional analyses. The third set of regressions was performed on 
these nine remaining variables and produced 4 variables that in combination best 
predict MCA-II reading and math scores across all three years. It is clear from the third 
set of regression equations that variables Ala, T6a, T21e and T16b_a are the best 
combination of predictors of student success across subjects, across grade levels and 
across years. 


In order to properly interpret these regression findings, the evaluation must first 
determine the direction of the correlations of each variable to the MCA-II scores. 
Appendix 3 displays Pearson correlations (see glossary in Appendix 1) between these 
four survey variables and MCA-II scores across subject and grades in 2008 that show 
the direction of the correlations. 


These correlations are very low, yet statistically significant due to the high N. However, 
what is more important about these correlations is the consistent direction of the 
correlations for each variable. All correlations for Ala and T16b_a are positive and all 
correlations for T6a and T21e are negative. The implication of this finding is that when 
Q Comp is implemented in schools the following set of conditions best predict or cause student 
achievement to increase: 


1. When school administrators feel that their teachers consider Q Comp to be 
successful in their school (Ala). 

2. When teachers feel that someone other than the principal is responsible for 
conducting Q Comp teacher evaluations/ observations T6a). 

3. When standards-based lessons are not the main topic of professional 
development activities and discussions, but other topics are addressed (T21e). 

4. When teachers feel that the addition of multiple career paths in their school 
will encourage them to remain in the teaching profession longer (T16b-a). 


(1) Relationship between number of years in Q Comp and student 
achievement 


Bivariate Pearson correlations enable us to understand the relationship between Q the 
number of years involved in Q Comp and student achievement. The correlations of 
years in Q Comp to MCA-II scores are presented in Appendix 3. All computations are 
for 2008. These correlations clearly indicate that there is a significant and positive 
relationship between the number of years a school is in Q Comp and student academic 
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achievement. This relationship is slightly less pronounced at the lower elementary 
level. 


(2) Charter and public schools 
Figure 12. Mean scale scores, reading and math, charter vs. public school district 
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Grade 10/11 Public SD 
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Figure 12 illustrates the differences between charter and public school districts in 2008. 
The t test results showing the mean differences between the two types of schools 
indicate that there are significant test score differences between charter and public 
schools. When comparing charter schools to public schools, there is consistency across 
subject areas and grade levels. In all cases examined, public school districts performed 
better on average than charter schools. The mean differences between charter and 
public school districts are statistically significant. 


(3) Student demographics 


Analysis of student demographic data is provided in Appendix 3. The demographic 
data demonstrates that white students comprise the vast majority of the student 
population in the database, followed by black students. There are 78.0 percent white 
students and 9.8 percent black students in the database. Also, the vast majority of 
students in the database are not LEP or Special Education students. Only 4.6 percent of 
the students are LEP and only 11.9 percent are special education. Also, a majority of 
students (76.1%) do not receive free or reduced lunch and, therefore, are not considered 
economically disadvantaged. 


D. DISCUSSION OF FINDINGS 


Our research activities and findings to date have revealed a number of interesting 
points that should only be considered in light of the limitations of our work and 
implications for the future of Q Comp. 


1. Limitations 


Our data and findings are somewhat limited by a number of factors. First, the brief 
duration of our evaluation created general circumstances that kept our evaluation 
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within the realm of descriptive more so than correlational or predictive research for a 
number of reasons: 


e With the exception of the analysis of MCA-II data described above, we were 
limited primarily to self-report data on perceptions of indicators and outcomes, 
rather than measurement of actual indicators and outcomes. Self-report data can 
tend to favor socially acceptable (i.e., positive or favorable) conclusions. 

e There was not enough time to conduct pre- and post-testing or conduct 
longitudinal research, both of which could have increased our statistical power. 

e There was little time to field test or otherwise vet our protocols and surveys with 
practitioners, though our instruments were approved by MDE. 

e We were only able to spend one day in each case study site, as we visited more 
schools than initially expected. 

e There was little time to assemble comprehensive participant contact lists in order 
to reach a broader group of practitioners and to collect data from them. 

e Some of the study participants who gave feedback to represent experiences with 
Q Comp have not been involved with the program for long (less than one year). 


We also understand that Q Comp implementation can take several years in order to 
‘trickle down’ to student-level change that could be detectable by tests such as the 
MCA-IIs. Repeated mixed measures deployed through multiple observations over time 
and involving carefully chosen comparison groups would be necessary to assure any 
causal relationship between Q Comp implementation and specific changes to student 
performance. In addition to the limitations associated with the brief duration of our 
evaluation, we are somewhat concerned with using MCA-II data as a measure of 
correlating aggregate student outcomes with Q Comp implementation. The advantages 
of using MCA-II data are that it is available across multiple schools, represents a 
common indicator of aggregate student performance over multiple years, and is 
designed to detect broad shifts in school improvement efforts. However, the MCA-II is 
not designed to measure the kinds of more sensitive changes (e.g., ‘value added’) in 
student performance that one might expect to be present as intermediate indicators of Q 
Comp’s impact on students. Ideally, a more sensitive assessment of student progress 
could be used for this purpose. 


2. Implications for the future of Q Comp 


Our evaluation also suggests several implications for the future of Q Comp. First, we 
recommend that MDE continues with a formative-summative, or possibly a summative 
evaluation in the future in order to address some of the limitations described above that 
stem from the short duration of our work. Ideally, this evaluation would be multi-year 
in order to maximize statistical power and the robustness of the research. Such an 
evaluation could utilize some of the current research questions, as well as additional 
questions that may arise from this evaluation or MDE’s further thought about Q Comp 
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implementation and impact. Additional data that should be analyzed to get a more 
complete picture of Q Comp implications would include voluntary and in-voluntary 
teacher turnover, recruiting statistics indicating that more candidates are attracted to Q 
Comp schools, etc. 


In addition to the mixed methods (including focus group interview protocols and 
surveys) that the Hezel team utilized since the fall of 2008, we suggest pilot testing and 
using the surveys that address some areas the current evaluation could not cover fully 
but could be of continued interest to MDE and Q Comp stakeholders, as well as other 
areas that we feel would benefit from further inquiry. 


The commendations and recommendations that follow more thoroughly describe the 
implications of our findings for the future of Q Comp, as we have learned from our 
evaluation research. 


Hezel Associates, LLC 49 


Q Comp Summative Evaluation Final Report for Public Release 


COMMENDATIONS AND RECOMMENDATIONS 


Here, we call upon the multiple data sources and findings described in the methods and 
findings sections above to present integrated commendations and recommendations for 
Q Comp. Each statement has been carefully crafted from a comprehensive and 
quantified approach to understanding all data collected during the fall of 2008 to the 
winter of 2009, and is organized by statewide and site-specific aspects of program 
implementation and impact. 


A. COMMENDATIONS 


1. Statewide implementation of Q Comp 


The perception of Q Comp’s impact on schools has overall been quite positive. 
Multiple data sources indicate there have been substantial changes to perceptions of 
school culture, instructional practices, schoolwide communication, focus, and 
management structure/ practices of Q Comp schools. 


Teachers, mentors, and coaches share a common view that the financial gains behind 
Q Comp are not a driving motivator for program participation. Feelings are 
somewhat mixed among teachers about whether the additional pay provided by Q 
Comp makes up for the additional work required. Even though teachers greatly 
appreciate the additional pay, they often feel as though doing smarter work that 
benefits students is simply part of their jobs. Teachers seem to feel that it would be 
inappropriate for someone in their profession to be motivated by money, while at the 
same time they indicate that basing this increased compensation on effort, as opposed 
to experience, is fair. 


Minnesota’s teachers seem to be more supported in their role of improving students' 
educational achievement since Q Comp’s inception, compared to non-Q Comp 
teachers, though we cannot link these trends statistically. Conversations among 
heterogeneous groups of practitioners are more frequent and more focused on teaching 
and learning than ever before. This support is particularly valuable for new teachers in 
their first few years in the profession. Practitioners appreciate that resources are 
invested in multiple types of intermediate contributors to improved student 
performance (e.g., professional development) and not just raw performance outcomes 
as measured by state or other standardized tests. 


Q Comp emphasizes all teachers growing professionally, which is consistent with 
MDE’s goals for the school improvement process. Despite different degrees of 
emphasis on job-embedded professional development in TAP and Q Comp schools, 
both have an increased focus on student needs and teacher accountability, which 
appear to be outcomes of all professional development models described by educators 
who participated in our research. Q Comp has prompted teachers, regardless of 
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experience level, content area, school type, geographic location or other factors often 
targeted by educational initiatives, to reflect on lesson execution and try new strategies. 
While administrators and teachers sometimes disagree on the execution and impact of 
their models, all generally agree that Q Comp has added real value and depth to their 
professional development opportunities. 


The flexible nature of Q Comp has been helpful to empowering local districts and 
schools with resources they can use to undergo the school improvement process. In 
Minnesota, each district can choose to pursue or not pursue Q Comp participation--it is 
not mandated. As districts apply, they may articulate plans and goals that are relevant 
to them, as long as the five components are addressed. MDE is not imposing how Q 
Comp should look, respecting that different districts have unique conditions and issues. 
Additionally, a district’s Q Comp goals may be updated each year and if a district 
decides Q Comp is not working for them, it can later be voted down. 


2. Districtwide implementation of Q Comp 


The perception that instructional practices have improved under Q Comp is 
widespread within Q Comp schools. Administrators and teachers within Q Comp 
schools cite an increased willingness to revise the curriculum when necessary, more 
deliberate construction of lessons, increased use of proven instructional strategies, and 
better instructional delivery since implementing Q Comp. Additionally, teachers 
reported using student data to drive instruction more and described how they set 
clearer, more effective instructional goals now that their school participates in Q Comp. 


Schools that have implemented Q Comp tend to view the program as an integration 
of each of the five components, and not simply a performance pay system. Indeed, 
the program does seem to be well-integrated, with all five elements reinforcing each 
other. However, non-Q Comp schools who have considered implementing Q Comp 
tend to view it as a vehicle for providing more money to teachers. 


Onsite and meaningful professional development that is integrated into teachers’ 
schedules under QO Comp facilitates teachers’ participation in regular and more 
substantive, as well as school improvement, activites. Most participating teachers 
believe that under Q Comp, professional development has become much more valuable 
and relevant to teachers. Q Comp seems to have embedded a sense of personal 
accountability, as teachers are expected to take what they learn back to their classrooms 
and follow up with the results. In fact, in at least two schools, classroom application of 
professional development is integral to the Q Comp evaluations. One of the most 
commonly described positive changes in teacher attitudes since schools joined Q Comp 
is a renewed focus on their own development and teaching practices. Onsite and 
worthwhile professional development opportunities appear to be less available at non- 
Q Comp schools. 
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Teachers are sharing and collaborating around student needs and instructional 
practices more than they ever have since Q Comp was implemented. Schools are 
taking a more collaborative approach to helping all students under Q Comp rather than 
each teacher being responsible for his/her own students, which results in students 
experiencing consistency in goals and approaches as they move to different classrooms 
throughout the day and progress through the grade levels. Additionally, some districts 
described the professional development activities associated with Q Comp as bringing 
teachers from different schools together and enabling them to share ideas and best 
practices on topics such as proven instructional strategies, better instructional delivery, 
and classroom management. Related to increased collaboration was a stronger sense of 
community in Q Comp schools where teachers described feeling more collegiality with 
other teachers and a sense that “we're in this together”. Q Comp gives teachers a 
common opportunity to get to know others beyond their own grade level, and to build 
relationships across the whole school. At Q Comp schools, teachers are sharing ideas 
during designated meetings, whereas at some non-Q Comp schools, teachers do not 
have time to share because they are too busy planning, grading, photocopying, etc. and 
the sharing that does occur is informal, such as in the hallway or lunchroom. 


Participating schools praise Q Comp for providing a unifying focus and framework 
for collaborating around instruction, planning and professional development. 
Schools reported having common performance goals that provide clear expectations for 
both teachers and students under Q Comp. One school reported that its schoolwide 
performance goals and focus were on student achievement in reading and math; this 
focus could be seen in teachers’ professional improvement plans, in teacher observation 
rubrics, and ultimately in performance pay. Another school reported that they had a 
unified focus on reading, with their teachers working more cohesively toward a single 
goal. These common goals, emphasized by Q Comp, provided a unity of purpose 
stemming from a collaborative, student-centered focus. 


Teachers attribute greater consistency in the way that expectations for students are 
set and in the teaching strategies that are being used to their participation in Q 
Comp. Under Q Comp, there is more consistency and cohesion in how teachers teach, 
and therefore in each student’s experience as (s)he travels from class to class. Almost all 
of the case study sites mentioned improved communication as a significant impact of Q 
Comp. This was discussed in the context of giving Q Comp administrators and 
teachers a common language/vocabulary to use (e.g., “there is a schoolwide continuum 
in which we talk using the same language”). Communication regarding students also 
improved, given the increased level of contact between teachers. Teacher meetings 
allow teachers to “know what specific students are struggling with” and then adapt 
accordingly. Some sites also mentioned a better flow of information from the 
administration to the teachers, because teacher leaders were a part of the school 
leadership team and also worked closely with non-leader teachers. Q Comp also 
provides a holistic structure that can unify other initiatives, causing schools to feel more 
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organized, and teachers to feel less as though there are too many things going on at 
once. 


Generally, observations in Q Comp schools are viewed as constructive, whereas in 
non-Q Comp schools, observations are perceived to be more evaluative. In Q Comp 
schools, teachers are more comfortable with administrators and other teachers 
observing them and providing feedback in large part because they felt these 
observations are non-judgmental and non-threatening. Observations are seen as an 
opportunity to learn and share their expertise with one another. While not all 
administrators may take this approach, those that portray the non-judgmental approach 
to teacher observations are having great success. 


In some Q Comp schools, there has been an overall shift from administrative 
decision making to teacher decision making, which administrators and teachers view 
favorably. Teachers described teacher leaders that are more accessible and 
approachable than administrators, while principals appreciate having qualified 
delegates to facilitate instructional leadership among a sometimes large teaching staff. 


Teachers from Q Comp schools are generally familiar with their career ladder 
systems, or believe this information is widely available and easily accessible. 
The ways in which the career ladder systems are implemented vary from school to 
school but most teachers believe the ladder system is fair and offers teachers equal 
opportunity (particularly people at younger ages) because the primary means of 
advancement is to take a position of greater responsibility. 


When Q Comp is implemented in schools, the following set of conditions best 
predict student achievement to increase: 


1. When school administrators feel that their teachers consider Q Comp to be 
successful in their school. 

2. When teachers feel that someone other than the principal is responsible for 
conducting Q Comp teacher evaluations/ observations. 

3. When standards-based lessons are not the main topic of professional 
development activities and discussions, but other topics are addressed. 

4. When teachers feel that the addition of multiple career paths in their school will 
encourage them to remain in the teaching profession longer. 


There is a significant and positive relationship between the number of years a school 
has been implementing Q Comp and student achievement and the number of years a 
school is in Q Comp with student academic achievement. This means as a school has 
more years of experience with Q Comp implementation, student achievement tends to 
increase. This relationship is slightly less pronounced at the lower elementary level. 
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B. RECOMMENDATIONS 


In addition to our commendations, we recommend the following: 


1. Statewide implementation of Q Comp 


Target audiences for Q Comp awareness campaigns should include both districts and 
schools. Our evaluation revealed a general lack of awareness and common 
misconceptions about Q Comp among non-Q Comp districts. Principals and teachers 
felt not enough was being shared with them at the school-level. Further, non-Q Comp 
schools described a lack of job-embedded professional development activities and a 
need for greater coherence in their schools, seemingly without realizing that Q Comp 
could provide resources to support these activities. We offer the following strategies as 
suggestions for addressing this recommendation: 


Strategy: Provide information about the variety of ways that districts and 
schools have designed programs to meet the requirements of specific elements 
of Q Comp. This could be done by showing examples of well developed 
applications. It would be particularly useful for teachers to understand the range 
of ways that Q Comp schools deliver incremental compensation to teachers, since 
there is a misconception that Q Comp schools link salary increases exclusively to 
test scores. The fact that professional development, teacher evaluations and 
schoolwide student achievement are also used to determine compensation may 
alleviate some of the concerns that non-Q Comp teachers have about the 
program. 


Strategy: Better utilize the Q Comp informational conference offered by MDE 
each January. This event goes largely unrecognized among teachers. Non-Q 
Comp schools perceive that MDE offers good conferences and workshops, but 
that the Department does not have a good advertising system. These events 
represent an outstanding opportunity to discuss Q Comp benefits and share best 
practices and ideas. Principals say they don’t see the information coming in and 
they “have to dig to find something that is offered.” 


Strategy: Provide information about what Q Comp does - and does not - 
involve. There are a number of different options for fulfilling the performance 
pay requirement of Q Comp, and MDE could consider developing examples of 
the most common models to give teachers an idea of the range of possibilities. 
This also means clarifying all the factors involved in the pay increase. Non- 
participating schools have concerns that Q Comp is all about student test scores. 
Urban school teachers in particular have reservations about financial rewards 
tied to a student population that is less likely to do well on tests or is transient. 
Convincing non-Q Comp schools of the value of the non-compensation aspects 
of Q Comp will be critical if more schools are to join the program. 
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Strategy: Encourage districts and schools that are interested in Q Comp 
participation to speak with or observe a school that is already successfully 
participating. This opportunity would provide interested schools with an 
explanation and visualization of how to “fit” Q Comp into teachers’ schedules to 
reduce fears about more responsibilities and demands on their time. 


Strategy: Highlight the benefits of Q Comp participation to school and district 
administrators. For example, Q Comp can provide rubrics for teacher 
observations and consistency in instruction. 


Strategy: Indicate that changes in culture and thinking can be positive. 
Movement away from the traditional Steps and Lanes approach to compensation 
can be difficult and time-consuming for teachers and administrators. Even in Q 
Comp schools, the traditional Steps and Lanes system still remains prominent in 
teachers’ minds - teachers still believe that years of experience and highest 
academic degree earned should play the primary role in determining base pay 
and a significant role in determining supplemental pay. 


Strategy: Increase publicly available success stories about Q Comp 
implementation and impact. For example, teachers at non-Q Comp schools are 
often times hearing about negative experiences with Q Comp, rather than 
positive experiences. Publicize the benefits of participation to minimize 
misconceptions and confusion. Teacher leaders at successful Q Comp schools 
would be outstanding spokesmen about the benefits and challenges associated 
with implementing Q Comp. Schools can be overwhelmed with having too many 
initiatives. If they want to pursue Q Comp, they need to know either how to 
integrate it or what else can be taken off their list of responsibilities. 


2. Districtwide implementation of Q Comp 


In addition to the many positive aspects of Q Comp implementation detected during 
the evaluation, there were also several areas where we can offer recommendations for 
districts and schools to facilitate an improved transition to or implementation of Q 
Comp. 


Monitor and respond to teachers’ experiences with Q Comp. One participating case 
study school gave an example of an in-house survey that is conducted each year to 
assess teachers’ opinions of how Q Comp is working in the school. Administrators use 
this data to make changes to meetings, agendas, and strategies. Paperwork for goal 
setting, for example, was seen as difficult or tedious, so this task has been modified. 


Show teachers how to manage the aspects of Q Comp that are viewed as 
cumbersome. This is a good practice for all schools to consider so teachers do not vote 
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Q Comp down when it is time for a re-vote. Nearly all case study schools noted an 
increased workload for teachers due to Q Comp - more meetings, paperwork, and 
requirements tax teachers’ available time. While there was some benefit in terms of 
time commitments with Q Comp, overall, it is fairly clear that the system requires more 
time of teachers than non-participating schools. The degree of comfort with this 
additional time required appears to be based on whether the teachers believe that they 
are being fairly compensated for their investments and how worthwhile they believe 
the investment of time is to them. Teachers often described taking on the additional 
responsibilities and worrying about how they will fit it all in later, which could lead to 
burnout. 


Support each district in clarifying how Q Comp complements other district 
initiatives. Because each district may have particular mandates, initiatives, issues and 
priorities, teachers may be confused or overwhelmed about how Q Comp fits in. 
Consider minimizing teacher fears and frustrations by articulating what each of the Q 
Comp acronyms stand for, how initiatives can be integrated and which ones take 
priority over others. 
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Appendix 1: 
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e TAP: Teacher Advancement Program. TAP is more rigorous than Q Comp, 
therefore, any TAP school in Minnesota is also viewed as a Q Comp school by 
MDE. TAP is only offered in two districts in the state (one public and one 
charter). TAP has four elements (multiple career paths, ongoing applied 
professional growth, instructionally focused observations and additional 
compensation for teachers). Within Minnesota, TAP has been implemented 
earlier (starting in the 2004-2005 school year) than any Q Comp school (first 
available in the 2005-2006 school year). An instructional rubric created by the 
National Institute for Excellence in Training is used in TAP. 


e Regression - A statistical model in which an outcome variable (or dependent 
variable) is predicted from one or more predictor variables (or independent 
variables). Regression assumes a causal relationship between the outcome 
variable and the predictor variables. Multiple regression is used when an 
outcome is predicted from a linear combination of two or more predictor 
variables. Stepwise regression is a method of multiple regression in which 
predictor variables are sequentially entered into the model (or equation) based 
on each predictor variable’s semi-partial correlation with the outcome variable. 


e Analysis of Variance (ANOVA) - A statistical test that determines whether group 
means differ significantly. A One-Way ANOVA is used if the groups are 
distinguished based on a single variable. 


e t-tests - A form of ANOVA in which the means of only two groups are tested to 
determine whether they differ significantly. Independent t-tests are used when 
the two groups are from independent samples. Dependent t-tests are used when 
the two groups are from the same sample. 


e Pearson Correlation - A measure of the strength of the linear relationship 
between two variables. A correlation coefficient can take any value from -1 to +1. 
A positive correlation means that as one variable changes, the other changes in 
the same direction. A negative correlation means that as one variable changes, 
the other changes in the opposite direction. A 0 correlation means that as one 
variable changes, the other doesn’t change at all. 
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Appendix 2: Summary of MDE focus group interview 
findings 
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Q Comp Focus Group Interviews 
Summary of Notes and Potential Implications for Project 


Observation 


Implications 


Difficulties in Implementation and Areas for 
Improvement 


Views differ about the most difficult 
component for districts to implement - pay 
and professional development were generally 
considered to be more difficult, though 
evaluations came up frequently as an area for 
potential improvement 

The career ladder seems to be the easiest 
component to implement, partially because 
some schools/ districts already had one in 
place 

It is felt that the components should be more 
integrated (e.g., comp and professional 
development being influenced by 
evaluations). There is a strong desire for Q 
Comp to be a comprehensive, integrated 
program, and not just a career ladder or a 
single performance bonus 


We will need to delve relatively deeply into the issue of 
evaluations - how they are working and how they could be 
improved 

Potential Survey questions 


O 


Are in-class teacher performance evaluations linked to the 
types of professional development that are focused on? 
Are evaluations linked to the level of performance pay 
received? 

A - Did your school already have a career ladder in place 
before Q Comp was implemented? 

How frequently are teachers evaluated in the classroom? 
How could teacher evaluations be improved (greater 
frequency, greater breadth of topics, more 

targeted / focused, evaluations done by other teachers, 
better training for evaluators, evaluations by a greater 
number of individuals)? 

How much time per week do teachers spend sharing ideas 
and collaborating with other teachers? 

Which approach most closely describes job-embedded 
Professional Development at your school: Outside experts 
are brought in to provide professional development 
content; Teachers within the school are tapped to share 
insights and ideas as professional development content; A 
combination of these approaches 
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Perceptions of Q Comp 


Perceptions of Q Comp have generally been 
positive from the teacher perspective. When 
teachers have not had control or input into the 
process, however, feelings can be more 
negative 

Principals are generally supportive, though 
some may view Q Comp more negatively 
because it means they lose a certain amount of 
administrative control 


We will need to examine whether there are differences in 
positive/negative feelings about Q Comp between teachers 
and administrators at a given site 
Potential Survey questions 
o Were teachers (i.e., all teachers or a subset) able to provide 
input about what the Q Comp system would look like? 
oO How were Principals involved in the design process? 


Measures of Success 


Some potential measures of success include: 
increased collaboration, increased interest in 
improving teaching practices, renewed 
excitement in teaching, improved school 
culture, increased focus on using student 
data, increased focus on student performance 
in general, school is more attractive to 
teachers we want to hire, more pay linked to 
academic achievement, better student 
performance 

We should also probe for potential 
unintended consequences 


Potential Survey questions 
0 To what degree do you believe that Q Comp has done the 
following: 


Increased teacher collaboration 

Created collaboration that has given me new ideas and 
teaching strategies to use in the classroom 

Improved relationships between teachers 

Changed school culture for the better 

Improved teacher morale 

Renewed my excitement in the career of teaching 
Increased my interest in improving my teaching 
practices 

Improved the performance of our students 

Increased focus on student performance as a measure 
of school and teacher success 

Increased focus on using student data in the classroom 
Made our school more attractive to teachers we might 
hire 

Reduced the number of good teachers who have left 
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our school to pursue other jobs or other teaching 
assignments 

« Helped new teachers to feel more supported in their 
first year 

" Helped new teachers to improve their performance in 
their first year 

* Caused the best teachers in my school to receive the 
highest level of rewards (e.g., bonuses and/or salary 
increases) 


Peer Review Process 
e The Peer review process is intended to 
examine how schools are implementing vs. 
the model and to provide schools with an 
opportunity to learn from one another. The 
focus group interviews seemed to be happy 
with the process 


e Potential Survey questions 
o A- Are you satisfied with the feedback you receive from 
the MDE Peer Review process? 
o A - How could the peer review process be more valuable 
from your perspective? 


e Potential Survey questions 
o A- Are you satisfied with the feedback you receive from 
the conference and networking meetings? 
o A- How could these sessions be more valuable? 


Conference and Networking Meetings 


e The conference and networking meetings 
enable attendees to learn from one another 
and share ideas 


Relationship to TAP 
e The components of the Q Comp system are 


modeled off of TAP, but they wanted more 
flexibility than TAP has to offer. They didn’t 
want to mandate the TAP approach 

e There are a few Q Comp schools that use the 
TAP system 


Potential Changes to System 
e Schools generally do a good job of 
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implementing - though there may be a few e May want to ask Principals whether they’d be receptive to 

bad apples, there is no need to impose more performance pay for themselves 

rules and structure because of them e We should also delve into how principals were involved in the 
e They are hesitant to make significant changes design process and how their role should change as a 

just yet. Potential structural changes include consequence of Q Comp? If principals’ roles need to change 

adding Principals or other school staff to significantly (e.g., shift from a “command and control” model 

performance pay systems and adding more to more of a “devolved leadership” model), how was this 

structure/ definition to the guidelines for communicated and what leadership training/support was 

performance pay provided? Are principals accountable for successful Q Comp 

implementation? 
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Appendix 3: Student Performance Data Tables 
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3. Predicting Student Achievement 


Stepwise regression reveals the combination of variables that together have a causal 
relationship with the variable that we wish to affect. Table 64 below illustrates the first 
iteration of regression using all 232 independent variables from the teacher and 
administrator surveys. The R coefficient is the multiple correlation coefficient and is 
cumulative down the list of variables, so the last variable in the list gives a total 
cumulative regression coefficient at the point where adding another variable adds 
nothing to the prediction. The corresponding R Sqaure is the percent of variance 
accounted for in MCA-II scores from this list of survey variables. The variables 
designated with an asterisk (*) in the below tables were also included in regression 
equations in either 2006 or 2007 or both. Each change in R Sqaure was tested for 
statistical significance using an F ratio. The contribution of each variable as it entered 
the equation was significant at p<.01. 


Table 12. Regression Iteration 1: Reading, Grade 3, 2008 


Variable R R square 
T6b 181 | .033 * 
A19a .225 | .05 # 
T16b a .242 | .059 * 
A2a .255 | .065 * 
T13e_sup | .267 | .071 
AQa .279 | .078 * 
A8c .288 | .083 * 
Al3a_2 .300 | .09 
A16b_b .314 | .099 + 
A6b .398 | .158 

Table 13. Regression Iteration1: Math, Grade 3, 2008 
Téa 164 | .027 + 
Ala .196 | .038 * 
A19a .213 | .045 
T2d .231 | .053 
A6e .252 | .063 * 
T22f .269 | .072 
T21f .292 | .085 ‘Z 
A2d .319 | .101 
T6e 526 | .276 


Table 14. Regression Iteration1: Reading, Grade 8, 2008 


Variable R R Square 


T6b .188 | .035 
T16b .235 | .055 
Alga 299 | .065 
A2a .273 | .075 
Al16a .308 | .095 
A8&c 325 | .105 
T20 425 | .181 
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Table 15. Regression Iteration 1: Math, Grade 8, 2008 


T21e .154 | .024 iW 
T21b .200 | .040 
A6c .228 | .052 
AQ9b .241 | .058 
Téa .248 | .061 
A4d .259 | .067 
A4c .269 | .072 
A16b_b .281 | .079 
A13b_2 -426 | .182 


Table 16. Regression Iteration 1: Reading, Grade 10, 2008 


Ala 150 | .022 
Téa .209 | .044 
A5h .240 | .057 
A21b .262 | .069 
T21e .275 | .076 
A7 .284 | .081 
T5) .291 | .084 
Adi .297 | .088 
T5d 302 | .091 
T22a .305 | .093 
T4c .308 | .095 
Al6b_b | .311 | .097 
A5j 333 | .111 
T19b 433 | .187 
Table 17. Regression Iteration 1: Math, Grade 11, 2008 

T8a 167 | .028 

Téa .205 | .042 

A4b .258 | .067 

T12 .273 | .075 

A6e .285 | .081 

T16c .300 | .090 


The second iteration of regression equations were conducted to determine if additional 
analyses from 2006 and 2007 would change the variable combinations that best predict 
student achievement. Table 18 illustrates how many regression equations in 2006 and 
2007 included which of the 11 variables from 2008 regressions. In this way we can 
eliminate some of the variables that predicted only in 2008 and not in the previous two 
years. The asterisks (*) in Table 18 indicate the number of regression equations in 2006 
and 2007 that included that particular variable. 
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Table 18. Regression Iteration 2: Inclusion of 2006 and 2007 


Variable 2008 2007 2006 
* 


Ai9a 

T6 a KKK * 
T21f 

T6b - * 
T16b_a * 
A2a * 

A9a 

A8c ail 
A16b_b a 

Ala 7 * 
T21e * * 


All 232 variables were entered into regression equations for both subjects and for three 
levels (grades 3, 8 and 10/11) for 2006 and 2007, just like they were for 2008. The 
results indicated that nine out of the 11 original variables from the 2008 analyses were 
also included in at least one regression equation in these additional analyses. 


The third iteration of regressions was performed on these nine remaining variables. 
Table 19 below shows the nine variables identified from the second iteration. The 
numbers in the table refer to the number of regression equations that included each 
variable as a contributor to MCA-II variance for each of the three years. Variables Ala, 
Toa, T21e and T16b_a were included in the greatest number of equations across the 
years and, therefore, are the best combination of predictors of student success across 
subjects, across grade levels and across years. 


Table 19. Regression Iteration 3: Number of Regression Equations by Year 


Number of 
Variable 2008 | 2007 2006 [Equations 
A19a 1 1 
Ala 2 4 4 10 
A2a 1 2 3 
Téa 3 2 4 9 
A8c 0 
T21e 2 4 5 11 
T16b_a | 2 2 2 6 
A16b_b 0 
T6b 0 


Table 20 below displays Pearson bivariate correlations between these four survey 
variables and MCA-II scores across subject and grades in 2008. This needed to be done 
to determine the direction of the correlation between the predictor variables and 
student achievement. The bivariate correlations in Table 20 are very low, yet 
statistically significant due to the high N. All correlations for Ala and T16B_a are 
positive and all correlations for T6b and T21a are negative. 
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Table 20. Correlation Matrix Between Final Predictor Variables and MCA-II 
Variable R,Grade3 M,Grade3 R,Grade8 M,Grade8 R,Grade10 M,Grade11 


T16b a 


11, 655 11, 131 12, 824 12, 547 13, 295 12, 471 


ar 


4. Relationship between Q Comp Years to Student Achievement 


Bivariate Pearson correlations enable us to understand the relationship between the 
number of years involved in Q Comp and student achievement. Table 21 illustrates the 
correlations of years in Q Comp to MCA-II scores. All computations are for 2008. 


Table 21. Correlation of Years in Q Comp and MCA-II 


Subject! Subject! 
Grade Correlation Pvalue N Grade Correlation Pvalue | N 
R/3 .093 .000** 16212 | M/3 .039** .000 15124 
R/4 .07 .0O0** 16141 | M/4 .058** .000 15231 
R/5 .06 .0O0** 16509 | M/5 .028** .0O1 15613 
R/6 .09 .000** 16693 | M/6 .O77** .000 15911 
R/7 .065 .000** 16943 | M/7 .072** .000 16227 
R/8 .083 .000** 17432 | M/8 .O6** .000 16704 
R/10 .05 .000** 17696 | M/11 .015* .048 16310 
*p<.05 
*D<.01 


5. Charter and Publc Schools 


Oneway ANOVAs and t-tests are used to test the mean differences between charter and 
public schools. Table 22 documents the differences between charter and non-charter 
schools in 2008. 


Table 22. Charter vs. Public School Districts 


Subject, Grade Public SD Charter Mean _ T value 


R, Grade 3 36531 35733 15.1** 
N 45816 1420 

M, Grade 3 35994 35409 17.0% 
N 44720 1351 

R, Grade 8 85493 84659 16.9** 
N 50688 847 

M, Grade 8 85277 84191 19.1** 
N 49925 727 

R, Grade 10 105600 _| 104500 24.5** 
N 51526 1017 

M, Grade 11 114300 __| 112200 32.1** 
N 48588 933 

*9<.01 
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6. Student Demographics 


The figures below show demographic breakdowns from the database by ethnicity, 
students with limited English proficiency, special education students, and students 
receiving free and reduced lunch. This data is reported by grade level for all three years 
- 2006, 2007, and 2008. 


Figure 13. Ethnicity 


45000 5 
40000 - = Am Indian 
35000 + mAsian/PI 
30000 - @ Hispanic 
25000 + Black 
20000 - m White 
15000 + 
10000 + 
5000 + 
ne oe oe oe ho Oe Oe 
Grade | Grade | Grade Grade | Grade | Grade | Grade | Grade 
3 4 5 6 7 8 10 11 
mAmIndian| 529 | 567 | 539 | 600 | 618 | 640 | 502 | 467 
mAsian/Pl | 3387 | 3332 | 3309 | 3364 | 3324 | 3335 | 3154 | 2870 
Hispanic | 2637 | 2479 | 2399 | 2378 | 2241 | 2200 | 1775 | 1370 
Black 5190 | 5089 | 5169 | 5178 | 5143 | 5084 | 4332 | 3623 
mg White 35487 | 35850 | 36863 | 37703 | 39002 | 40246 | 42742 | 41202 
Figure 14. Limited English Proficient 
60000 + mLEP 
50000 - m Not LEP 
40000 + 
30000 + 
20000 + 
10000 + 
0 
Grade | Grade | Grade | Grade | Grade | Grade | Grade | Grade 
3 4 5 6 7 8 10 11 
m LEP 3327 | 2973 | 2680 | 2563 | 2280 | 2143 | 1593 | 830 
Not LEP | 43903 | 44344 | 45599 | 46660 | 48048 | 49362 | 50912 | 48700 
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